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EXECUTIVE SUMMARY 



The Even Start Family Literacy Program was established in 1989 (P.L. 107- 
110, Sec. 1231) to help break the cycle of poverty and illiteracy for low-income families, 
by improving the literacy skills of parents and their young children (U.S. Department of 
Education 2003). Even Start projects offer family literacy services, defined as four 
integrated instructional components (P.L. 107-110, Sec. 9101 (20)): 

• Early Childhood Education (ECE); 

• Parenting Education (PE); 

• Parent-Child Literacy Activities (PC); and 

• Adult Education (AE). 

Two previous studies of the Even Start Program showed that parents and 
children who participated in Even Start did not have better literacy outcomes than 
parents and children in a randomly assigned control group that did not receive Even 
Start services. The Even Start Classroom Literacy Interventions and Outcomes (CLIO) 
Study is the third randomized study of Even Start. As opposed to the earlier evaluations 
that investigated the effectiveness of Even Start relative to randomly assigned control 
groups in which parents and children were not enrolled in Even Start, the CLIO study 
was intended to intervene by offering the combination of research-based, literacy- 
focused early childhood education and parenting education curricula (the "CLIO 
combined curricula"). The CLIO study was intended to determine (1) whether the CLIO 
combined curricula were more effective than existing Even Start instructional services, 
and (2) whether research-based parenting education curricula that focus on child 
literacy (the "CLIO parenting curricula") added value to research-based, literacy- 
focused early childhood education curricula (the "CLIO preschool curricula"). 

This report presents 2-year impacts of the CLIO curricula on child language, 
literacy, and social competence; parenting skills; parent literacy; and instructional 
practices and participation in preschool and parenting classes. 
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Main Findings 



The main findings from the CLIO impact analyses are that (1) the CLIO 
combined curricula had statistically significant, positive impacts on some of the 
hypothesized precursors to the development of children's early literacy skills, including 
instructional supports for literacy, child social competence, and parenting skills; but (2) 
the CLIO combined curricula did not have statistically significant impacts on any of the 
child language development and early literacy outcomes. 

The CLIO combined curricula had statistically significant positive impacts on 

• two of five measures of preschool instruction: support for print 
knowledge and literacy resources in the classroom; 

• one of three measures of parenting instruction: the amount of parenting 
education time spent on child literacy; 

• both measures of parenting outcomes: parent interactive reading skill 
and parent responsiveness to their child; and 

• child social competence. 

The CLIO combined curricula did not have statistically significant impacts 
on: 



• three of five measures of preschool instruction: support for oral 
language, support for phonological awareness, and support for print 
motivation; 

• two of three measures of parenting instruction: the amount of parenting 
education time spent on parenting skills not related to child literacy or 
the amount of parent-child time spent with parents and their children 
interacting on child literacy activities; 

• monthly hours of preschool instruction received by children or monthly 
hours of parenting instruction received by parents; 

• parent English reading skills (includes vocabulary); and 
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child expressive language (in English or Spanish), receptive vocabulary, 
phonological awareness (Elision or Blending), print knowledge, or 
syntax and grammar. 



The CLIO parenting curricula added value to the CLIO preschool curricula 
by increasing significantly the amount of parenting education time spent on child 
literacy, the amount of parenting education time spent on parenting skills not related to 
child literacy, and parent interactive reading skill. The CLIO parenting curricula did not 
significantly add value to the CLIO preschool curricula with respect to parent 
responsiveness, child literacy outcomes, or child social competence. 



Background 

The Even Start Lamily Literacy Program provides grants to local projects to 
provide family literacy services to low-income families. Lamily literacy services are 
defined as the integration of the four instructional services mentioned above with 
sufficient intensity in terms of hours and duration to make sustainable changes in a 
family. An important premise underlying the Even Start program is that the 
combination of early childhood education, parenting education, parent-child literacy 
activities, and adult education adds value to participant outcomes. That is, language 
and literacy outcomes for children in Even Start should be improved directly, through 
the effects of participation in preschool, and indirectly, through enhancements in both 
parenting skills and parent literacy. Parenting skills are expected to be enhanced 
through participation in parenting and parent-child activities, and parent literacy 
through participation in adult education literacy training. 

Since the inception of Even Start in 1989, the U.S. Department of Education 
has sponsored three national evaluations of the program that focused on performance 
and effectiveness. Two of the three national evaluations included experimental studies 
that randomly assigned eligible and interested families to participate in Even Start or a 
control group of families who would delay participation in Even Start for at least 1 year 
(St.Pierre et al. 2003; St.Pierre et al. 1995). The results of these studies showed that Even 
Start projects were not effective at improving the literacy skills of participating 
preschool-age children and their parents. That is, literacy gains made by Even Start 
parents and children were no different from literacy gains made by control parents and 
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children. The control group for these randomized studies was composed of parents who 
wanted to enroll their children in Even Start but who were randomly assigned to 
participate in Even Start in the year following the evaluation. About two-thirds of these 
control parents were unable to arrange any other formal early childhood education 
(ECE) services during the period of the evaluation, so the control condition mostly 
corresponded to at-home care by parents or extended family members (St.Pierre et al. 
2003, p. 162). 

The absence of significant effects of Even Start on literacy skills, along with 
new requirements in the reauthorized Even Start legislation to base instruction on 
scientifically based reading research (Sec. 1231 (2)(D)), prompted an examination of the 
Even Start model to determine how it could be improved. The lead investigators of the 
most recent national Even Start evaluation (St.Pierre, Ricciuti, and Rimdzius 2005) 
addressed several questions about Even Start's apparent ineffectiveness: (1) whether the 
Even Start model was fully implemented, (2) whether Even Start's instructional services 
were sufficiently intensive, (3) whether Even Start families participated sufficiently, and 
(4) whether the quality of Even Start's instruction and curriculum content was sufficient 
to lead to positive effects. 

The CLIO study was, therefore, designed to test the extent to which research- 
based, literacy-focused curricula strengthen Even Start services and lead to significant 
impacts on parents and children. 1 Specifically, the CLIO study was designed to address 
two primary research questions: 

• Is the combination of research-based, literacy-focused preschool, 
parenting, and parent-child curricula (the CLIO combined curricula) 
more effective than the existing combination of services in Even Start? 

• Do research-based parenting and parent-child curricula (the CLIO 
parenting curricula) that focus on child literacy add value to the CLIO 
preschool curricula? 



1 This is consistent with Even Start's second legislative evaluation requirement (Sec. 1239 (2)), which is to 
identify effective programs that can be duplicated and used in providing technical assistance. CLIO is 
also consistent with the requirement for research (Sec. 1241) that examines successful family literacy 
services. 
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Thus, the study was an evaluation of the incremental effectiveness of 
providing the CLIO curricula to Even Start projects. 



CLIO Study Design and Curricula 

Through a competitive process, the CLIO study selected two combined 
preschool and parenting education curricula, 2 each of which were based on the most 
current research on the development of children's early literacy skills. CLIO used these 
curricula in four combinations — two that implemented the combined research-based 
preschool and parenting curricula and two that implemented the research-based 
preschool curricula in combination with existing parenting education services. The 
CLIO study used an experimental design in which 120 Even Start projects were 
randomly assigned to implement one of the four CLIO curricula combinations or to be 
in a control group that provided their regular pre-CLIO instructional services (see table 
ES-1). 



The CLIO combined curricula and CLIO preschool curricula were 
implemented in the sample of Even Start projects during program years 2004-2005 and 
2005-2006. Implementation included summer training sessions for project directors and 
teachers in each year, as well as ongoing support for preschool and parenting education 
staff from the curriculum developers over the 2-year period. 

The CIRCLE group at the University of Texas-Houston Health Sciences 
Center teamed with Abrams & Company Publishers to provide the Let's Begin with the 
Letter People preschool curriculum to CLIO. Let's Begin is a preschool curriculum that 
builds early literacy skills and uses 26 imaginary characters that represent the letters of 
the alphabet. The CIRCLE group provided the Play and Learning Strategies (PALS) 
parenting curriculum to CLIO. PALS focuses on responsive parenting and teaches 
parents techniques to build their children's language and cognitive development. 



2 The study team dedded not to include Even Start's adult education component in the test of research- 
based curricula because (1) most projects provided a variety of adult education services at different 
levels (adult basic education (ABE), general equivalency diploma (GED), English as a second language 
(ESL)) to meet family needs, (2) a substantial portion of projects used community service providers to 
deliver adult education services, and (3) the research on effective adult education models is still in its 
infancy. 
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Table ES-1. Specification of the Five CLIO Study Groups 



Even Start 

instructional 

component 


Study group 


Group 1 


Group 2 


Group 3 


Group 4 


Group 5 


LET'S BEGIN 
with the Letter 
People (ECE) 

CLIO preschool 
curriculum 


LET'S BEGIN with 
the Letter People and 
Play and Learning 
Strategies (PALS) 
(ECE/PE) 

CLIO combined 
curriculum 


Partners for 
Literacy (ECE) 

CLIO preschool 
curriculum 


Partners for 
Literacy 
(ECE/PE) 

CLIO combined 
curriculum 


Control 


Early 

childhood 

education 


LET'S BEGIN 


LET'S BEGIN 


Partners for 
Literacy 


Partners for 
Literacy 


As usual 


Parenting 

education 


As usual 


PALS 


As usual 


Partners for 
Literacy 


As usual 


Parent-child 
joint literacy 
activities 


As usual 


PALS 


As usual 


Partners for 
Literacy 


As usual 


Adult 

education 


As usual 


As usual 


As usual 


As usual 


As usual 



NOTE: Shaded areas identify instructional components that were provided by the CLIO curriculum developers. 



The University of North Carolina at Chapel Hill provided the Partners for 
Literacy curriculum to CLIO. The preschool Partners curriculum is based on game-like 
activities conducted with pairs of children and instructional strategies designed to 
support children's cognitive and language development. The parenting Partners 
curriculum adapts the game-like activities and instructional strategies from the 
preschool curriculum and trains parents to use these with their children at home. The 
Partners curriculum also includes training in problem-solving skills for children and 
parents. 



CLIO Contrasts 

As discussed earlier, the CLIO study addressed two key research questions: 

1) Is the combination of research-based, literacy-focused preschool, 
parenting, and parent-child curricula (the CLIO combined curricula) 
more effective than the existing combination of services in Even Start? 





2) Do research-based parenting and parent-child curricula (the CLIO 
parenting curricula) that focus on child literacy add value to the CLIO 
preschool curricula? 

The first research question was addressed analytically by combining projects 
that received the CLIO combined curricula (study groups 2 and 4 in table ES-1) and 
comparing their outcomes with those of control projects (study group 5). The study's 
second research question was addressed analytically by combining projects that 
received the CLIO combined curricula (study groups 2 and 4), and comparing their 
outcomes with those of projects that received the CLIO preschool curricula (study 
groups 1 and 3). 



CLIO Data Collection and Outcome Constructs 

The study team collected data over a 3-year period. The first year of data 
collection was 2003-2004, prior to implementation of the CLIO curricula. The second 
and third years of data collection (2004-2005 and 2005-2006) corresponded to the two 
CLIO curricula implementation years. 3 

The study team conducted the following types of data collection in all CLIO 
projects: direct assessments of child language and literacy; teacher ratings of child social 
competence; videotapes of parent-child interactions; interviews of parents; direct 
assessments of parent literacy; observations of classroom instruction in preschool, 
parenting education, and parent-child classes; surveys of teachers and project directors; 
and tallies of child and parent participation in instructional services. The study team 
also observed and rated the fidelity of implementation of the CLIO curricula. The 
outcome constructs used in the CLIO impact analyses are presented in table ES-2. 



3 The CLIO study is also following children into kindergarten and first grade. 




Table ES-2. CLIO Outcome Measures 





Outcome 


Data collection 
instrument 


Mode of data 
collection 


Domain 


CHILD 


1 


Expressive language: English 


Individual Growth and 
Development Indicator 
(IGDI) 


Child assessment 


Emergent 

literacy 


2 


Expressive language: Spanish 


3 


Receptive vocabulary 


Peabody Picture 
Vocabulary Test (PPVT) 


4 


Phonological awareness: Elision 


Comprehensive Test of 
Phonological and Print 
Processing (Preschool - 
CTOPPP) 


5 


Phonological awareness: Blending 


6 


Print knowledge 


7 


Syntax and grammar 


Test of Language 
Development (TOLD-3) 


8 


Social competence 


Teacher rating form 


Teacher rating 


Socio- 

emotional 

development 


PARENT 


9 


Parent interactive reading skill 


Read Aloud Together 
Profile & Parent 
Interview 


Video observation, 
parent report 


Parenting 

skills 


10 


Parent responsiveness 


11 


Reading & vocabulary skill 


Parent assessment 
battery 


Parent assessment 


Parent 
language & 
literacy 


INSTRUCTIONAL 


12 


Support for oral language 
development 


Observation Measures of 
Language and Literacy 
Instruction (OMLIT) 
and Parenting Education 
and Child/Parent 
Observation (PECAP) 


Classroom 

observation 


Preschool 

classroom 

instruction 


13 


Support for print knowledge 


14 


Support for phonological 
awareness 


15 


Support for print motivation 


16 


Literacy resources in classroom 


17 


Parenting education time spent on 
child literacy 


Parenting 

classroom 

instruction 


18 


Parenting education time spent on 
parenting skills 


19 


Parent-child time spent interacting 
on child literacy activities 


Parent-child 

classroom 

instruction 


20 


Child: Monthly horns of preschool 
instruction received 


Instructional Services 
Participation Form 
(ISPF) 


Project report 


Participation 

amount 


21 


Parent: Monthly hours of parenting 
and parent-child instruction 
received 
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Implementation of the CLIO Curricula 



Fidelity to Planned CLIO Curricula. Fidelity of implementation to the CLIO 
curricula in the sample projects was rated both by independent observers and by the 
curriculum developers. Both sets of ratings indicated that, on average, implementation 
of the CLIO combined curricula and the CLIO preschool curricula only reached about 
50 percent of full implementation. Fidelity ratings for the Let's Begin and PALS projects 
were generally higher than those for the Partners for Literacy projects, for both the 
preschool and parenting classrooms but particularly for preschool classrooms. Most of 
the average fidelity ratings by observers and developers were higher in 2006 than in 
2005 with the exception of observer ratings for Partners for Literacy preschool 
classrooms. 

Exposure to the CLIO Curricula. Participants (parents and children) in any 
intervention need a minimum level of exposure to the curriculum to obtain the 
hypothesized benefits. Even Start guidelines do not specify an expected level of 
exposure for children or parents, and the hours of instruction offered by local projects 
vary widely. In each implementation year, while projects reported that they offered 
preschoolers an average of 80 hours of preschool education per month, children in 
CLIO projects actually participated in preschool an average of 50 hours per month. 
Parents also received only partial exposure to the parenting curricula. Projects reported 
that they offered parents an average of 25 hours of parenting education and parent- 
child activities per month, but parents participated for an average of 13 hours of 
parenting education and parent-child activities per month. These levels of participation 
relative to hours of services offered are in line with what was documented in previous 
Even Start evaluations (St.Pierre et al. 2003, p. 129). 

Control Projects. Project directors reported that about 75 percent of the CLIO 
control projects used a formal early childhood curriculum (most often High/Scope or 
Creative Curriculum), and about 60 percent used a formal parenting curriculum (most 
often locally developed). Observations of control classrooms showed that they spent 
about 45 percent of the day in activities that are often considered by developmental 
psychologists to have particularly high value for children because of the opportunities 
for children to construct knowledge and receive feedback on their interactions with 
materials, peers, and adults in the classroom (Bruner and Watson 1983). The remainder 
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of the control group day was spent in daily group activities including review of the 
calendar/weather/attendance, gross motor play and transition, and meals/snacks. 



Impact Findings 

Impacts of the CLIO Combined Curricula. The study showed that Even 
Start projects assigned to the CLIO combined curricula did not exhibit better child 
language and literacy outcomes than Even Start projects assigned to the control group 
(figure ES-1). In the figures in this section, effect sizes for the combined curricula are 
indicated by filled diamonds (relative to the control group) and open circles (relative to 
the preschool curricula), and 95 percent confidence intervals 4 are shown as horizontal 
bands on either side of the diamond or circle. Effect size indicates the difference in 
outcome between the average subject who received the treatment and the average 
subject who did not. 5 

There were no statistically significant impacts of the CLIO combined 
curricula on any of the seven measures of child language and literacy skills (six in 
English and one in Spanish), as can be seen by the fact that none of the confidence 
bands exclude zero, even before adjustment for multiple comparisons. Estimated effect 
sizes on emergent literacy outcomes were all smaller than 0.13 in absolute value, with 
confidence interval limits all bounded by 0.27 in absolute value. However, the CLIO 
combined curricula did have a statistically significant positive effect on child social 
competence (behavior in class) as rated by preschool teachers. The effect size of the 
impact of the CLIO combined curricula on child social competence was 0.22. 



4 The confidence intervals may be interpreted as follows. If the experiment were to be independently 
repeated a very large number of times under the same general conditions, drawing on the same 
population of schools and students, and on every repetition both an effect estimate and a confidence 
interval on that estimate were calculated, then, over the long run, 95 percent of the confidence intervals 
would contain the long-run average of estimated effects. 

5 Effect size was calculated by taking the difference between the treatment and control group means and 
dividing that difference by the standard deviation of the control group's scores in 2005. 




Figure ES-1. Effect Sizes for CLIO Combined Curricula on Child Outcomes Relative 
to Both the Control Group and the CLIO Preschool Curricula (average of 
spring 2005 and spring 2006) 
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The CLIO combined curricula had a statistically significant positive impact 
on both of the parent outcomes examined (figure ES-2). The effect size of the impact on 
parent interactive reading skill was 0.48, and the effect size of the impact on parent 
responsiveness to their child was 0.22. Even though CLIO did not manipulate adult 
education curricula, the study assessed parent reading skills and vocabulary and 
showed that the CLIO combined curricula did not have a statistically significant impact 
on these skills (figure ES-2). 

Figure ES-2. Effect Sizes for CLIO Combined Curricula on Parent Outcomes Relative 
to Both the Control Group and the CLIO Preschool Curricula (average of 
spring 2005 and spring 2006) 
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The CLIO combined curricula had a statistically significant positive impact 
on two of five measures of instructional support for literacy development in preschool 
classrooms (figure ES-3). The effect sizes of the statistically significant impacts on 
support for print knowledge and literacy resources in the classroom were 0.69 and 0.52, 
respectively. There was no statistically significant impact on the following three 
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Figure ES-3. Effect Sizes for CLIO Combined Curricula on Instructional Outcomes 
Relative to Both the Control Group and the CLIO Preschool Curricula 
(average of spring 2005 and spring 2006) 
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preschool instructional measures: support for oral language development, support for 
phonological awareness, or support for print motivation. 6 

The CLIO combined curricula had a positive impact on one of the three 
measures of parenting education and parent-child classroom instruction (figure ES-3). 
The effect size of the impact on the amount of parenting education time spent on child 
literacy was 1.01. There was no statistically significant impact on the amount of 
parenting education time spent on parenting skills not related to child literacy or the 
amount of parent-child time spent with parents and their children interacting on child 
literacy activities. 

The study also examined whether the CLIO combined curricula had an 
impact on participation levels (figure ES-4). The results showed that there was no 
statistically significant impact of the CLIO combined curricula on either child levels of 
participation in preschool or parent levels of participation in parenting education or 
parent-child activities. Neither of the confidence bands exclude zero. 

Added Value of the CLIO Parenting Curricula. CLIO parenting curricula 
did not add significantly to the effectiveness of the CLIO preschool curricula on any of 
the seven measures of child literacy skills or on child social competence (figure ES-1). 
That is, adding research-based parenting components focused on child literacy did not 
add significantly to children's outcomes beyond what was achieved with the CLIO 
preschool curricula. (In figures ES-1 through ES-4, the effect sizes for the added value of 
the CLIO parenting curricula are indicated by open circles.) The estimated effect sizes of 
the CLIO parenting curricula on emergent literacy outcomes were all smaller than 0.11 
in absolute value, with confidence interval limits all bounded by 0.23 in absolute value. 

However, the CLIO parenting curricula did have a statistically significant 
positive incremental effect on parent interactive reading skill (effect size of 0.30) (figure 
ES-2). The difference on parents' responsiveness to their child between the CLIO 
combined curricula and the CLIO preschool curricula, while similar in size to the 
statistically significant difference between the CLIO combined curricula and the control 
group, was not statistically significant. 



6 Although the confidence bands for support for phonological awareness and support for print 
motivation exclude zero, the effect sizes are not significant once adjusted for multiple comparisons. 




Figure ES-4. Effect Sizes for CLIO Combined Curricula on Participation Relative to 
Both the Control Group and the CLIO Preschool Curricula (average of 
spring 2005 and spring 2006) 
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There were statistically significant incremental effects of the CLIO parenting 
curricula on two of the instructional measures. The effect sizes of the incremental effects 
of the CLIO parenting curricula on the amount of parenting education time spent on 
child literacy and the amount of parenting education time spent on parenting skills not 
related to child literacy were 0.68 and -0.45, respectively (figure ES-3). There was no 
statistically significant incremental effect of the CLIO parenting curricula on how time 
was spent in parent-child classes or (as expected) in preschool classes. 

Finally, the CLIO parenting curricula did not have a statistically significant 
incremental effect on child participation in preschool or on parent participation in 
parenting education (figure ES-4). 
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Secondary Analyses 



Three secondary analyses were conducted to examine the variation in 
impacts of the CLIO curricula. 

Year of Implementation. One hypothesis of the CLIO study was that 
impacts might be greater in the second year, when most projects could be assumed to 
have had 2 years to reach full implementation. With respect to child outcomes, there is 
evidence that the CLIO combined curricula had statistically significant negative effects 
on four of the seven children's language and literacy outcomes in the first year of 
implementation. By the second year, rough parity with the control group was achieved. 
There is little evidence of differential effects by year for child social competence, parent 
outcomes, instructional outcomes, and participation. 

Analysis of Growth for Child and Parent Outcomes. While the primary 
impact analysis measures parent and child outcomes at the end of preschool, the study 
also examined impacts on the pattern of growth from fall to spring. The only significant 
finding was that the CLIO parenting curricula had a positive incremental effect on 
parent responsiveness to their child. 

Interactions of Study Group with Ethnicity and Home Language. About 
half of all children in the CLIO sample spoke a home language other than English. An 
analysis of interactions found that impacts on children's emergent literacy did not vary 
significantly as a function of home language or ethnicity. 



Summary 



Prior studies have established that Even Start does not have statistically 
significant impacts on children's emergent literacy or on parent literacy. The CLIO 
study investigated whether the implementation of research-based, literacy-focused 
curricula would improve literacy outcomes for Even Start children and parents. 
Although there were positive impacts on some of the literacy supports in preschool 
classrooms, on time spent on child literacy in parenting education classes, on parenting 
skills, and on children's social competence, there were no statistically significant 
impacts on children's language and literacy. There was no evidence that the failure to 
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find impacts on these core outcomes was due to a lack of fidelity in the treatment 
classrooms or cross-over in the control classrooms. 
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1 . 



BACKGROUND 



In this chapter, we discuss the Even Start program, the purpose of the 
current study, and the organization of the remainder of the report. 



The Even Start Program 

The Even Start Family Literacy Program was established in 1989 with the 
goal of improving the academic achievement of low-income young children and their 
parents, especially in the area of reading (U.S. Department of Education 2003). Even 
Start projects offer four integrated instructional activities for low-income families: 

• age-appropriate early childhood education to prepare children for 
success in school and life experiences (early childhood education, or 
ECE); 

• training for parents regarding how to be the primary teacher for their 
children (parenting education, or PE); 

• interactive literacy activities between parents and their children (parent- 
child literacy activities, or PC); and 

• parent literacy training that leads to economic self-sufficiency (adult 
education, or AE). 

The underlying premise of Even Start, and of the family literacy model more 
generally, is that these four instructional components are necessary for improved child 
literacy and are maximally effective when integrated into a unified program. That is, 
child language and literacy should be improved directly, through participation in ECE, 
and indirectly through improvements in both parenting skills and parent literacy. 
Parenting skills are expected to be improved through participation in PE and PC 
activities, and parent literacy through participation in AE. Also, Even Start services 1 are 
to be of "sufficient intensity in terms of hours, and of sufficient duration, to make 
sustainable changes in a family." 



1 Family literacy services are defined in Sec. 203 of Title II of The Workforce Investment Act of 1998, Public 
Law 105-220, also known as the Adult Literacy and Family Education Act of 1998. 
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Even Start Has Not Performed Up To Expectations 



Since 1989, the U.S. Department of Education (ED) has sponsored three 
national evaluations of the Even Start program that focused on performance and 
effectiveness. Two random assignment studies that were part of these evaluations 
(St.Pierre et al. 1995; St.Pierre et al. 2003) showed that Even Start projects were not 
effective at improving the literacy skills of participating preschool-age children and 
their parents. That is, literacy gains made by Even Start parents and children were no 
different from literacy gains made by control parents and children. The control group 
for these randomized studies was composed of parents who wanted to enroll their 
children in Even Start but who were randomly assigned to participate in Even Start in 
the year following the evaluation. About two-thirds of these control parents were 
unable to arrange any other formal ECE services during the period of the evaluation, so 
the control condition mostly corresponded to at-home care by parents or extended 
family members. 2 

The absence of significant effects on literacy skills prompted an examination 
of the Even Start model to determine how it could be improved. The lead investigators 
of the most recent national evaluation of Even Start (St.Pierre, Ricciuti, and Rimdzius 
2005) addressed several questions that might explain Even Start's apparent 
ineffectiveness: (1) whether the Even Start model was fully implemented, (2) whether 
Even Start's instructional services were sufficiently intensive, (3) whether Even Start 
families participated sufficiently, and (4) whether the quality of Even Start's instruction 
and curriculum content was sufficient to lead to positive effects. 

Data from the national evaluation showed that Even Start projects were, 
indeed, able to fully implement the program, and that Even Start projects offered 
instructional services at a level of intensity that is comparable to mainstream programs 
offering the individual parts of a family literacy program. However, the evaluation also 
documented that Even Start families participated at low levels and for a relatively short 
period of time. Further, evaluation data showed that Even Start's instructional services 
were not of uniformly high quality. In particular, the national evaluation found that (1) 
Even Start's early childhood education programming was not of higher quality than the 
instruction received by control children and was not of higher quality than the 



2 See page 162 of St.Pierre et al. (2003). 
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instruction received by Head Start children, (2) the adult education programs provided 
to Even Start parents varied widely in their quality and the extent to which they focused 
on literacy, and (3) the parenting education programs offered by Even Start projects 
were similar in content and delivery systems to mainstream parenting programs, which 
have little research evidence on the extent to which they are effective at enhancing 
either parenting skills or child literacy. On the basis of these findings, the evaluators 
suggested that one promising avenue for improving Even Start would be to increase the 
extent to which Even Start's instructional services focus on literacy. 



The CLIO Study: Seeking to Improve Even Start 

The Even Start Classroom Literacy Interventions and Outcomes (CLIO) 
Study is the third randomized study of Even Start. Prior studies have investigated the 
effectiveness of Even Start relative to control groups in which parents and children were 
not enrolled in Even Start. In contrast, the CLIO study examined the effectiveness of 
four different curricular packages against the "regular" Even Start program. These 
curricular packages featured research-based literacy instruction. Two of the packages 
focused solely on early childhood education instruction, while the other two packages 
combined instruction in early childhood education with instruction in Even Start's two 
parenting components. This approach is supported by the strengthened mandate of 
Even Start from the Literacy Involves Families Together Act ( LIFT 2001) and the No Child 
Left Behind Act ( NCLB 2001), which call for Even Start projects to provide 

• high-quality, intensive instructional programs, 

• instructional programs based on scientifically based reading research, 
and 

• reading readiness activities based on scientifically based reading 
research. 

In addition, the CLIO study is consistent with Even Start's second legislative 
evaluation requirement, which is "to identify effective Even Start programs . . . that can 
be duplicated and used in providing technical assistance to Federal, State, and local 
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programs. 3 The CLIO design is also consistent with the research goals and methods 
with respect to the components of successful family literacy services: 4 

Preschool and parenting instruction were manipulated to construct four 
distinct experimental curricula: 

• Two CLIO combined curricula that focused on child literacy both in 
preschool and parenting instruction. These were each a combination of a 
CLIO preschool curriculum and a CLIO parenting curriculum. 

• Two CLIO preschool curricula that had an intense focus on child 
literacy but left parenting instruction alone. These were each a 
combination of a CLIO preschool curriculum and whatever approach to 
parenting instruction was already in use at the Even Start projects. 

In keeping with Even Start's four-component family literacy approach, all of 
the Even Start projects participating in the CLIO study continued to provide AE to 
parents, but the AE instruction was not changed as part of the CLIO study. 

The CLIO study addresses two primary research questions: 

1) Is the combination of research-based, literacy-focused preschool, 
parenting, and parent-child curricula (the CLIO combined curricula) 
more effective than the existing combination of services in Even Start? 

2) Do research-based parenting and parent-child curricula (the CLIO 
parenting curricula) that focus on child literacy add value to the CLIO 
preschool curricula? 

Thus, CLIO is an evaluation of the incremental effectiveness of providing 
these research-based literacy-focused instructional services, over and above the existing 
instruction provided by Even Start projects. 



3 Evaluation goal #2 under Sec. 1239 of the No Child Left Behind Act of 2001, Public Law 107-110. 

4 Sec. 1241 of the No Child Left Behind Act of 2001, Public Law 107-110. 
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The conceptual model for the CLIO study (figure 1-1) builds on the premise 
that Even Start improves child outcomes both directly (through ECE participation) and 
indirectly (through parenting and adult education). The oblongs on the left hand side of 
the model illustrate the instructional services that CLIO sought to improve (ECE, PE, 
and PC). The next three boxes show that the CLIO curricula are hypothesized to 
improve the instructional practices of staff working with Even Start children and 
parents. Improved instructional practices are hypothesized to lead to short-term 
enhancements, by the end of preschool, in children's development and in parent 
behaviors and skills. Improvements in parenting skills also were hypothesized to 
enhance children's development. 5 The AE instruction provided to Even Start parents 
was not changed as part of the CLIO study; however, the family literacy model assumes 
that AE produces improvements in parents' literacy and educational levels, which 
contribute to enhanced child development. 6 This relationship is represented in the three 
boxes along the bottom of the model. 



The model includes longer term impacts, since the hypothesis is that changes 
in children's development and skills by the end of preschool will result in improved 
reading and language skills in the early school grades. Longer term changes in 
parenting skills and in parents' improved literacy and education are hypothesized to 
support these improved child outcomes. IES is interested in exploring whether there is 
support for these hypotheses, and so is conducting a follow-up study to address the 
question: Do the CLIO curricula produce positive effects at the end of kindergarten or at 
the end of first grade? 7 



5 Although Even Start serves families with at least one child between birth and age 8, and most projects 
serve children throughout this entire age range, the study focused on preschool-age children and their 
families. At the time the study was designed, the Even Start program office focused on school readiness 
by attempting to improve the effectiveness of educational services for 3- and 4-year-olds. CLIO did not 
include infants and toddlers because we felt that (1) there was no conclusive evidence that formal 
instruction in language and literacy is helpful for that age group, (2) there was an absence of systematic 
curricula for children in this age group, and (3) there was only limited information about how services 
for infants and toddlers were administered in Even Start. CLIO did not include school-age children 
since Even Start's role for school-age children is predominantly one of coordination with public schools. 
It was deemed unlikely that Even Start projects would be able to bring about curriculum changes in the 
public schools. 

6 The study team decided not to include Even Start's adult education component in the test of research- 
based curricula because (1) most projects provided a variety of adult education services at different 
levels (ABE, GED, ESL) to meet family needs, (2) a substantial portion of projects used community 
service providers to deliver adult education services, and (3) the research on effective adult education 
models is still in its infancy. 

7 Followup data collection with children in kindergarten and first grade is being conducted, and findings 
from that data collection will appear in a later report. 
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Figure 1-1. CLIO Model of Effects for Even Start Projects 
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Organization of the Report 



In the remainder of this report we fully describe the research design (chapter 
2), provide a description of the CLIO curricula (chapter 3), discuss the methods for the 
analyses (chapter 4), and present the findings of our analyses (chapters 5 through 7). 
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2 . 



RESEARCH DESIGN 



In this chapter, we describe the selection and implementation of the CLIO 
curricula, recruitment and random assignment, the CLIO projects at baseline, the data 
collection schedule and methods, the sample design, and the development of the 
outcome measures. 



Selection of the CLIO Curriculum Developers 

The hypothesis underlying the CLIO study is that an increased focus on 
literacy in preschool and parenting instruction would improve parent and child 
outcomes for Even Start families. To select interventions that were literacy focused and 
based on research, a public process was used in which developers of preschool and 
parenting curricula were invited to submit proposals for review by an expert panel. The 
Request for Proposals was prepared, and proposals were solicited in spring 2003 from 
curriculum developers. Eight proposals were received, and the authors of the four 
highest rated proposals were invited to make oral presentations to the expert panel. The 
expert panel rated the proposals on several key criteria, including the quality of the 
proposed intervention, capability of the institution to meet the requirements of the 
study and bring the interventions to scale, and staff qualifications and experience. 

The key criterion (worth 55 of the 100 possible points) related to the quality 
of the proposed intervention. Under this criterion, the proposals were judged on the 
extent to which the content of the interventions (both preschool and parenting 
components) focused on literacy (specifically the domains of oral language, 
phonological awareness, print recognition and conventions of print), was appropriate 
for the Even Start population, and integrated the preschool and parenting components. 
The proposals were also judged on the evidence that the interventions were effective 
(particularly with populations similar to Even Start). 

The selection was based primarily on the ratings of the intervention content, 
specifically whether the content proposed was linked — either by previous research or 
expert judgment — to the study's targeted outcomes. Although the four highest rated 
proposals thoroughly documented their curricula's grounding in the research literature 
on emergent literacy, rigorous evidence of effectiveness was not extensive. After the 
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oral presentation round of the selection process, the expert panel judged three of the 
four remaining proposals as acceptable for inclusion in the study. Two proposals were 
chosen from the three based on the strength of the parenting component of the 
intervention. 

The two curriculum developers that were selected each has a preschool 
curriculum coupled with an integrated parenting curriculum. The developers were 
responsible for implementing their curricula in two modes: (1) preschool only, and (2) 
preschool and parenting combined. The curricula tested in CLIO were the following: 

• Let's Begin with the Letter People® and Play and Learning Strategies 
(PALS) — The CIRCLE group at the University of Texas-Houston Health 
Sciences Center teamed with Abrams & Company Publishers to develop 
and implement: 

(1) Preschool only: Let's Begin with the Letter People, a preschool 
curriculum that is built around 26 imaginary characters that 
represent the letters of the alphabet. Let's Begin was augmented with 
teacher training from CIRCLE on effective practices in early literacy. 

(2) Preschool and parenting combined: Let's Begin was linked with 
the Play and Learning Strategies (PALS) parenting curriculum. PALS 
was developed by CIRCLE for parents whose children are at risk for 
developmental delay and academic failure due to poverty, low 
family literacy, and other risk factors and teaches parents to 
understand where their child is on the developmental continuum 
and what techniques they can use to build their children's language 
skills, cognitive development, and school readiness. 

• Partners for Literacy — The University of North Carolina at Chapel Hill, 
Frank Porter Graham Child Development Center, developed and 
implemented: 

(1) Preschool only: ECE Partners for Literacy, a preschool curriculum 
based on game-like activities and interactive book reading conducted 
with pairs of children and designed to promote language 
development and emergent literacy. The curriculum is based on an 
earlier version that was used in the Abecedarian project. 
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(2) Preschool and parenting combined: ECE/PE Partners for Literacy, 
which links the preschool curriculum with an integrated parenting 
education curriculum that uses many of the same activities, 
strategies, and materials as the preschool curriculum. 

Effectiveness and Appropriateness of Selected Curricula 

Let's Begin and PALS. At the time of selection. Let's Begin had been 
implemented and field tested in preschool classrooms with populations similar to Even 
Start across the country. The developers cited the positive results of an ongoing efficacy 
study conducted by Abrams & Company, the curriculum publisher. The curriculum has 
take-home materials available in Spanish, and teacher materials contain suggestions for 
adapting activities for English language learners. PALS was developed for 
disadvantaged families and had been implemented in both English and Spanish. The 
developer cited positive results from a recent randomized controlled experimental 
study (Landry, Smith, and Swank 2003). Most PALS materials are available in both 
English and Spanish. 

Partners for Literacy. Partners for Literacy materials were developed for 
children from low-income families and at the time of selection had been used in 
preschool settings with populations similar to Even Start. Developers cited positive 
results from three randomized, controlled longitudinal research studies: the 
Abecedarian Project (Ramey et al. 1976), project CARE (Wasik, Ramey, Bryant, and 
Sparling 1990), and the Infant Health and Development Program (Ramey et al. 1992). 
Professionals and parents received materials for cultural responsiveness, and all 
materials for parents are available in English and Spanish. Teacher training also 
included time and materials devoted to teaching English language learners. 



Establishment of the Five Study Groups for the CLIO Study 

One hundred twenty Even Start projects were recruited and randomly 
assigned to one of five study groups: one of the four CLIO curricula or an "as is" control 
group that provided their regular pre-CLIO instructional services (table 2-1). Even 
Start's four instructional components define the rows of the table, while the four CLIO 
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Table 2-1. Specification of the Five CLIO Study Groups 



Even Start 

instructional 

component 


Study group 


LET'S BEGIN 
with the Letter 
People (ECE) 

CLIO preschool 
curriculum 


LET'S BEGIN with 
the Letter People 
and Play and 
Learning 
Strategies 
(PALS)(ECE/PE) 

CLIO combined 
curriculum 


Partners for 
Literacy (ECE) 

CLIO preschool 
curriculum 


Partners for 
Literacy 
(ECE/PE) 

CLIO combined 
curriculum 


Control 


Early 

childhood 

education 


LET'S BEGIN 


LET'S BEGIN 


Partners for 
Literacy 


Partners for 
Literacy 


As usual 


Parenting 

education 


As usual 


PALS 


As usual 


Partners for 
Literacy 


As usual 


Parent-child 
joint literacy 
activities 


As usual 


PALS 


As usual 


Partners for 
Literacy 


As usual 


Adult 

education 


As usual 


As usual 


As usual 


As usual 


As usual 



NOTE: Shaded areas identify instructional components that were provided by the CLIO curriculum developers. 



experimental groups and a control group define the columns. Shaded table cells 
identify instructional components that were provided by the CLIO curriculum 
developers. The CLIO study groups have the following characteristics: 

• Study Group 1: Even Start projects were assigned the Let's Begin 
preschool curriculum. These projects provided their usual PE, PC, and 
AE instructional services. 

• Study Group 2: Even Start projects were assigned both the Let's Begin 
preschool curriculum and the PALS parenting curriculum. These 
projects provided their usual AE instructional services. 

• Study Group 3: Even Start projects were assigned the Partners for 
Literacy preschool curriculum. These projects provided their usual PE, 
PC, and AE instructional services. 

• Study Group 4: Even Start projects were assigned both the Partners for 
Literacy preschool curriculum and the parenting curriculum. These 
projects provided their usual AE instructional services. 
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Study Group 5: Even Start projects assigned to the control group 
provided each of the four instructional components as usual. This was 
an "as is" or "business as usual" control group. 



Implementation of the Curricula 

Once selected, the curriculum developers modified their existing curricula as 
appropriate, developed materials, prepared professional development and 
implementation plans, and piloted their curricula with a small number of Even Start 
projects in 2003-2004. Curricula were revised on the basis of the pilot test, and plans 
were made for large-scale implementation. In summer 2004, a 4-day centralized training 
session was held for each curriculum, attended by project directors and teachers from 
the assigned Even Start projects. In addition, the curriculum developers provided each 
participating Even Start project with on-going support and technical assistance over the 
life of the study. 

Implementation of the CLIO curricula in Even Start classrooms began in the 
2004-2005 school year, and during that year, ongoing support was provided to each 
project through telephone calls and on-site visits. In summer 2005, a second centralized 
training was held to retrain Even Start staff who had been trained in the previous 
summer and to provide an initial training to new staff members. Each of the four CLIO 
curricula was then implemented for a second year during 2005-2006, again with 
ongoing support of the curriculum developers. The implementation phase of CLIO 
concluded at the end of the 2005-2006 school year. 



Design of the Study to Address the Research Questions 

The CLIO study was designed to address two primary research questions: 

• Research Question 1: Is the combination of research-based, literacy- 
focused preschool, parenting, and parent-child curricula (the CLIO 
combined curricula) more effective than the existing combination of 
services in Even Start? 
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Research Question 2: Do research-based parenting and parent-child 
curricula (the CLIO parenting curricula) that focus on child literacy add 
value to the CLIO preschool curricula? 



The first question was addressed analytically by combining projects in the 
second and fourth columns of table 2-1 (those assigned to the CLIO combined 
curricula), and comparing their outcomes with those of control projects (the fifth 
column). The second question was addressed analytically by combining projects in the 
second and fourth columns of table 2-1, and comparing their outcomes with those of 
projects in the first and third columns (those assigned to the CLIO preschool curricula). 

In addition to these two primary research questions, the CLIO study 
examined several secondary questions: 

• Instructional Practices: To what extent are particular preschool 
instructional practices associated with better child outcomes? 

• Parenting Practices: To what extent are parenting practices associated 
with better child outcomes? 

• Fidelity of Implementation: How much variation was there in the 
faithfulness with which CLIO projects implemented the assigned 
curricula? Were child and parenting outcomes better in projects with 
higher fidelity to their assigned curriculum? 

• Participation: To what extent is participation associated with better 
outcomes? 



Recruitment of Even Start Projects 

Recruitment of Even Start projects for the CLIO study began with a careful 
screening of projects to determine which ones met the study's eligibility requirements. 
To be eligible for CLIO, an Even Start project had to 

1. serve preschool children in a center-based instructional setting, 

2. enroll a minimum of either five 3- and 4-year-olds in one center-based 
classroom, or eight 3- and 4-year-olds in two center-based classrooms. 
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3. provide at least 12 hours per week of center-based preschool instruction, 

4. serve a majority of families who speak either English or Spanish, 

5. be able to exert control over the curricula used in preschool classrooms, 
and 

6. be willing to meet the study requirements, including being randomly 
assigned to one of the five study groups. 

Exerting control over preschool curricula was an eligibility criterion because 
Even Start requires projects to build on existing services, where possible, to avoid 
duplication. Thus, many projects do not directly provide all of Even Start's instructional 
services, but rather, coordinate with other programs to provide some services. For 
example, during the latest national evaluation (St.Pierre, Ricciuti, and Rimdzius 2003), 
22 percent of Even Start 3- and 4-year-olds who participated in center-based ECE 
received these services from Head Start programs. Projects that outsourced their 
instructional services in this way were not excluded from participating in CLIO, but few 
chose to do so, since the Even Start grantee often did not have control over the 
preschool curriculum. Hence, most CLIO projects were ones that provided their own 
preschool instruction. The study team did not, however, screen for eligibility based on 
(1) serving children and their parents in a center-based setting for the provision of 
parent-child activities or parenting education or (2) providing a specified number of 
hours per week of parent-child activities or parenting education instruction. 

In spring 2003, 1,150 Even Start projects operated throughout the United 
States. Telephone calls were attempted with 1,127 of these projects, excluding 23 
projects in Alaska, Hawaii, and Puerto Rico that were defined as ineligible due to the 
distances and prohibitive costs that would be associated with data collection. Telephone 
surveys were completed with 967 (86 percent) of the 1,127 Even Start projects in eligible 
states. Of these, 637 were ineligible to participate in the study for reasons such as not 
serving a sufficient number of preschool-age children, not offering preschool-age 
children at least 12 hours per week of center-based preschool instruction, serving 
primarily families that spoke languages other than English or Spanish, or not having a 
center-based ECE program. 
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During the screening process, we relaxed the requirements to broaden the 
eligibility pool by allowing (1) the enrollment of fewer numbers of children in center- 
based classrooms and (2) the provision of at least 10 hours (lowered from 12) of center- 
based preschool instruction. At the conclusion of the screening process, 330 Even Start 
projects were deemed eligible. 

Of the eligible projects, 120 were willing to participate in the study and were 
randomly assigned to one of the five study groups. (See figure 2-1 for the flow of the 
projects through the recruitment process.) 

CLIO was designed and implemented as a real-world study of literacy 
focused, research-based curricula in Even Start settings. Nationwide, Even Start settings 
vary widely on every aspect of the program. Even Start projects are mandated to offer 
instructional services that include early childhood education, adult literacy education, 
parenting education, and structured literacy interaction between parents and their 
children. They also are required to avoid duplication of services by building on existing 
community resources such as local adult education programs or Head Start. Even Start 
projects decide on the frequency and duration of instruction, whether instruction is 
primarily center-based or home-based, and whether to invent educational curricula 
from scratch, use published curricula, or use a hybrid of approaches. Based on the 
availability of local instructional services and the extent to which those services are 
perceived to be of high quality, project staff decide which activities will be supported by 
Even Start funds and which will be provided by collaborating agencies. Projects offer 
screening and referral services such as referrals for mental health counseling, services to 
battered family members, employment services, and screening or treatment for 
chemical dependency. Projects also offer support services such as transportation, 
flexible scheduling, childcare, nutrition assistance, health care, and meals to help 
families participate in the program. The CLIO curricula were implemented in a sample 
of 120 Even Start projects that were spread over 33 states and reflected this diversity. As 
a result, CLIO could not be a tightly controlled lab oratory- type experiment. 
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Figure 2-1. Flow of Even Start Projects Through CLIO Recruitment and 
Randomization 




NOTE: 

All groups had 24 projects providing at least 1 year of data for the combined spring 2005 and spring 2006 analysis. 
"One of the original 1 20 projects dropped out of the study prior to the first implementation year (2004-2005) and was 
replaced for 2005-06. 

**Six projects lost funding for the second implementation year (2005-2006) and had data for 2004-2005 only. 

***One project lost funding prior to the first implementation year (2004-2005) and was replaced for 2005-2006. 
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Random Assignment of CLIO Projects 



The CLIO random assignment plan focused on ways to minimize pre- 
existing differences among the five study groups. Before random assignment, 24 strata 
were formed, each containing exactly five projects. The variables used to form the strata 
were (1) size of project (number of 3- and 4-year-olds served), (2) proportion of children 
who were Spanish speakers, (3) year that the project was up for recompetition, and (4) 
region. (See table 2-2 for the cut points for each of these stratification variables.) 8 

Table 2-2. Stratification Variables Used in Random Assignment 



Stratification variables 


Categories 


Size of project 


3 categories: 

• large, defined as more than 28 3- and 4-year-olds 

• medium, defined as 10 through 28 3- and 4-year- 
olds 

• small, defined as 9 or fewer 3- and 4-year-olds 


Proportion of Spanish-speaking 
children 


3 categories: 

• very large, defined as more than 25 percent 
Spanish speakers 

• medium, defined as 8 percent through 25 percent 
Spanish speakers 

• small, defined as less than 8 percent Spanish 
speakers 


Last year of current grant 


3 categories: 

• 2003-2004 school year 

• 2004-2005 school year 

• other 


Region 


4 standard Census categories 



The highest priority was placed on size of project. The large category 
contained only 13 projects. Among these, we generally found either a very large 
proportion or a very small proportion of Spanish speakers, so within it we created just 
two strata, substratified only on percentage Spanish while ignoring the other two 
variables. The other three large projects were then mixed in with the medium projects. 



Simply crossing these four variables would have created 108 strata, more than could be used. 
Accordingly, extensive collapsing of preliminary strata was required. Because of the requirements of 
exactly five projects per stratum, we also sometimes had to make small changes in the thresholds. 
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Within the medium and small categories, we were able to use more categories of 
percentage Spanish and to pay some attention to the other variables. Once the 24 strata 
were finalized, the five projects in each were randomly assigned to the five study 
groups in early 2004. 

A comparison of the resulting five groups of projects (mostly in terms of 
variables collected at the spring 2004 baseline) showed that the random assignment 
plan resulted in well-matched study groups. There were no statistically significant 
differences among the five groups on 55 of 58 spring 2004 variables that were examined 
(see tables A-l through A-5 in appendix A). 9 Additionally, there were no statistically 
significant differences among the five groups on 23 of the 25 variables examined in 
spring 2005 and spring 2006 (see table A-l) 10 



Study Projects at Baseline 

The voluntary nature of participation and the eligibility criteria for the study 
meant that the CLIO sample was not nationally representative of Even Start projects. 
However, the 120 recruited projects were located in 33 states in all regions of the 
country and varied on characteristics such as population density, number of families 
served, percentage of families who are English language learners, and number of years 
as Even Start projects. In this section, we present descriptive statistics on the study 
sample, both to set a context for the evaluation and to provide a basis for assessing the 
study's external validity. 

Race/Ethnicity. In spring 2004, prior to implementation of the CLIO 
curricula, 57 percent of CLIO children were identified as Hispanic. In 2000-2001 (the 
most recent period with national data), 46 percent of Even Start parents nationally were 
Hispanic (table 2-3). Although the CLIO sample is not nationally representative, the 



9 Two of these three variables were included among the covariates chosen for the impact analysis. See 
chapter 4 for a full list of covariates. 

10 Two types of tests were used for testing for baseline balance across the study groups. For baseline item 
response theory (IRT) scores, hierarchical linear modeling (HLM) was used with a two-level setup 
(project and child) and no covariates other than strata. For all other types of variables, a stratum- 
adjusted Kruskal- Wallis test was run on project-level averages. No weights were used at the project 
level. Multinomial variables like race were transformed into a series of binary recodes, each of which 
was tested separately. 
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movement from 46 percent to 57 percent continues a long trend of increases in the 
percentage of Hispanic families served by Even Start. 

Table 2-3. Percentage Distribution of CLIO Children and Even Start Parents by 
Race/Ethnicity 



Even Start 





CLIO children 
spring 2004 


parents 

2000-01 


Race/ethnicity 

White 


22 


30 


Black 


11 


19 


Hispanic 


57 


46 


Other 


9 


5 



NOTE: Detail may not sum to totals due to rounding. 



SOURCES: U.S. Department of Education, Even Start Classroom Literacy Interventions and Outcomes Study, 
"Parent Interview," Spring 2004; U.S. Department of Education, Planning and Evaluation Service, Elementary and 
Secondary Education Division, Third National Even Start Evaluation: Program Impacts and Implications for 
Improvement, Washington, DC: 2003. 



Maternal Education. As of spring 2004, 38 percent of CLIO children had 
mothers with a high school diploma, GED, or higher (table 2-4). In 2000-2001, 15 percent 
of new Even Start parents had this level of education. Additionally, in 2003-2004, 24 
percent of new Even Start participants had a high school diploma, GED, or higher. The 
national statistics are based on parents from newly entering Even Start families, 
whereas the CLIO statistic is based on information about mothers from all Even Start 
families in each project, some of whom many have increased their education attainment 
as a result of participating in Even Start. Since prior research has shown that Even Start 
has a positive impact on GED attainment, it is not surprising that education attainment 
based on all families in the program, including those that have participated for many 
months, would be different from educational attainment based only on newly entering 
families. Of course, there may be other reasons as well for the high educational 
attainment of CLIO mothers relative to national Even Start figures. 

Hours of Instruction. Amount of instruction offered and received is a 
statistic that was calculated in prior Even Start studies. The CLIO projects are similar to 
the 2000-2001 national sample in terms of participation in PE and PC activities. CLIO 
parents participated in PE and PC activities for an average of 10 hours a month in 
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Table 2-4. Percentage Distribution of Educational Attainment for Mothers of CLIO 
Children and New Even Start Parents 





Mothers of 
CLIO children 
spring 2004 


New Even 
Start parents 
2000-2001 


New Even 
Start parents 
2003-2004 


Educational attainment 








HS, GED, or higher 


38 


15 


24 


Without HS/GED 


62 


84 


76 



NOTE: Detail may not sum to totals due to rounding. 

SOURCES: U.S. Department of Education, Even Start Classroom Literacy Interventions and Outcomes Study, 
"Parent Interview," Spring 2004; U.S. Department of Education, Planning and Evaluation Service, Elementary 
and Secondary Education Division, Third National Even Start Evaluation: Program Impacts and Implications for 
Improvement, Washington, DC: 2003; U.S. Department of Education, Consolidated State Performance Report 
School Year 2004-05. 



2003-2004, similar to the 2000-2001 national estimate of 11 hours a month (table 2-5). In 
2003-2004, projects in the CLIO study offered preschool-age children an average of 84 
hours of instruction each month, fairly similar to the national statistic of 76 hours a 
month in 2000-2001 (table 2-5). Children in the CLIO sample participated in preschool 
instruction an average of 42 hours a month during 2003-2004. Although this was only 
half of the amount offered to them, it nevertheless is greater than the national Even 
Start average of 33 hours a month of participation in preschool education in 2000-2001 
(table 2-5). 



Teacher Education. In spring 2004, 82 percent of the lead preschool teachers, 
22 percent of the preschool aides, and 89 percent of the lead PE teachers in the CLIO 
sample had an associate's degree or higher (table 2-6). 

Classroom Instruction. At baseline most CLIO projects (71 percent) reported 
that they used at least one formal curriculum in their preschool classrooms. These 
projects cited a wide range of instructional programming, including published 
comprehensive curricula, literacy-focused supplemental curricula, state curriculum 
frameworks, skills assessments linked to instructional strategies, informal non- 
published curricula, and local or other reading initiatives. Of the projects that used a 
formal curriculum, close to 70 percent reported using either High Scope or Creative 
Curriculum. About half (55 percent) reported that they used at least one formal 
curriculum for their parenting education sessions. Of those, about 40 percent used 
Parents as Teachers. Most projects (78 percent) reported that they did not use any 
formal curriculum in their parent-child interactive sessions. 
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Table 2-5. Average Monthly Hours of ECE Instruction Offered and Received and 
Average Monthly Hours of PE/PC Instruction Received, for CLIO and 
Even Start 





CLIO 

2003-2004 


Even Start 
2000-2001 


Projects 1 

Hours of ECE instruction offered per month 


84 


76 


Children 1 

Hours of ECE instruction received per month 


42 


33 


Parents 

Hours of parenting education and parent-child 
joint activities received per month 


10 


11 


1 Even Start hours for 2000-2001 represent instruction for 3- and 4-year-olds. 

SOURCES: U.S. Department of Education, Even Start Classroom Literacy Interventions and Outcomes Study, 
"Instructional Services Participation Form, and Project Director Survey," Spring 2004; U.S. Department of 
Education, Planning and Evaluation Service, Elementary and Secondary Education Division, Third National Even 
Start Evaluation: Program Impacts and Implications for Improvement, Washington, DC: 2003 



Table 2-6. Educational Attainment of Even Start Staff in CLIO Projects: Spring 
2004 



Staff position 


Percent 


Lead preschool teacher 
Educational attainment 




Less than associate's degree 


18 


Associate's degree or higher 


82 


Preschool aide 




Educational attainment 




Less than associate's degree 


78 


Associate's degree or higher 


22 


Lead PE teacher 




Educational attainment 




Less than associate's degree 


11 


Associate's degree or higher 


89 


SOURCE: U.S. Department of Education, Even Start Classroom Literacy Curricula and Outcomes Study, "Staff 
Survey," Spring 2004. 
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At baseline, all CLIO projects were observed with the Early Childhood 
Environment Scale-Revised Edition (ECERS-R; Harms, Clifford, and Cryer 1998) to 
assess the quality of the classroom environment, including use of space, materials and 
experiences to enhance children's development, schedule, and supervision. Specifically, 
the ECERS-R assesses 37 items that cover the six classroom areas: space and furnishings; 
personal care routines; oral language and reasoning skills; fine motor, gross motor, and 
creative activities; interactions among children and between children and staff; and 
program time and structure. Each item is ranked on a scale of 1 (inadequate conditions) 
to 7 (excellent conditions), with 3 representing minimal conditions and 5 representing 
good conditions. 

Across the 37 items, CLIO projects scored an average of 4.88 on the ECERS-R 
measure, suggesting that, on average, CLIO projects have good classroom 
environments. No project received an "inadequate conditions" rating. Approximately 
70 percent of projects received a rating of 4 or 5. These data suggest that CLIO 
classrooms are comparable to Head Start preschool classrooms. According to the Head 
Start Family and Child Experiences Surveys (FACES), the average overall ECERS-R 
score for Head Start classrooms was 4.91 in spring 2001 and 4.81 in fall 2003 (U.S. 
Department of Health and Human Services 2006). 

Even Start Continues to Work With a Needy Population. The data 
presented in this section show that Even Start families in the CLIO sample continue to 
face many of the same difficulties that were identified in the Third National Even Start 
Evaluation (St.Pierre et al. 2003). Income and education levels are low compared to most 
of America: 58 percent of Even Start families in the CLIO projects have monthly income 
below $1,500, and only 38 percent of Even Start mothers in the CLIO projects have a 
high school diploma. Furthermore, adult English fluency is poor. In spring 2004, 59 
percent of CLIO parents report a native language other than English. Among the non- 
native speakers, just 7 percent claim to speak and understand English very well 
(compared to 88 percent of the native speakers). Literacy is also poor. Overall, just 40 
percent of CLIO parents claim to read English very well. These self-reports of low 
fluency and literacy are borne out by the assessments. In particular. Even Start parents 
in the CLIO projects scored quite low on the Peabody Picture Vocabulary Test 
(PPVT) — 70 on average using publisher norm scores with a national mean of 100 and a 
national standard deviation of 15. 
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Given the unexpectedly large number of CLIO mothers with a high school 
diploma, GED, or higher educational attainment (relative to the most recent available 
national Even Start figures), we also looked at the breakdown of published PPVT scores 
by mother's education. We found that even among mothers with higher educational 
attainment, receptive vocabulary scores were very low: mothers who attended some 
college had an average PPVT score of 84, those with a high school diploma averaged 77, 
and those without a high school diploma averaged 63. 



Data Collection Schedule and Methods 

In this section, we discuss the CLIO study's data collection schedule and 
provide an overview of the data collected by data collection cycle (see table 2-7). 

The CLIO study's data collection schedule was as follows: 

• 2003-2004: baseline year. We collected baseline data on 3- and 4-year- 
olds and their parents in all Even Start projects participating in the CLIO 
study during the 2003-2004 project year. We also observed classroom 
instruction and collected information about the project in the spring of 
the baseline year. 

• 2004-2005: first year of implementation. We collected data on 3- and 4- 

year-olds and their parents in all Even Start projects participating in the 
CLIO study during the 2004-2005 project year. We also observed 
classroom instruction and collected information about the project in the 
spring of the first implementation year. 

• 2005-2006: second year of implementation. We collected data on 3- and 
4-year-olds and their parents in all Even Start projects participating in 
the CLIO study during the 2005-2006 school year. We also observed 
classroom instruction and collected information about the project in the 
spring of the second implementation year. 

Data were collected from (1) preschoolers (3- and 4-year-olds), (2) their 
parents, (3) classrooms, and (4) projects. Here we briefly describe the types of data 
collected and the methods for collecting these data. In the next section, we discuss the 
creation of outcome measures based on these data. 
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Table 2-7. Overview of Data Collection 





Data collection 
instrument 


Baseline year 


First implementation 
year 


Second 

implementation 

year 


Fall 2003 


Spring 

2004 


Fall 2004 


Spring 

2005 


Spring 2006 


Preschoolers 


Assessment 


Child Assessment 
Battery 


X 


X 


X 


X 


X 


Social-emotional 


Teacher-Child 
Rating Form 


X 


X 


X 


X 


X 


Videotape 


Read Aloud 
Together Profile 






X 


X 


X 


Participation 


Instructional 

Services 

Participation Form 


X 


X 


X 


X 


X 


Parents 


Assessment 


Parent Assessment 
Battery 


X 


X 


X 


X 


X 


Interview 


Parent Interview 


X 


X 


X 


X 


X 


Videotape 


Read Aloud 
Together Profile 






X 


X 


X 


Participation 


Instructional 

Services 

Participation Form 


X 


X 


X 


X 


X 


Classrooms 


Observation of instruction 


Observation 

Protocols 




X 




X 


X 


Observation of fidelity 


Observation 

Protocols 








X 


X 


Survey 


Teacher Survey 




X 




X 


X 


Projects 


Survey 


Project Director 
Survey 




X 




X 


X 



Data Collected From Preschoolers 



At each data collection cycle, trained field staff administered a battery of one- 
on-one child assessments. The battery covered the following domains: a test of 
expressive language, in both English and Spanish; a test of receptive vocabulary; two 
tests of phonological awareness. Elision and Blending; a test of print knowledge; and a 
test of syntax and grammar. These assessments are described in detail later in this 
chapter (see section entitled child outcomes). The study team conducted a week-long 
training prior to each data collection cycle to prepare field staff for data collection. 
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Assessments were conducted at each Even Start project, in a setting provided by the 
Even Start project staff. 

Each preschooler's teacher was asked to complete a Teacher-Child Rating 
(TCR) form at each data collection cycle. The TCR captured information on each child's 
behavior and social skills. More information about the TCR is provided later in this 
chapter (see section entitled child outcomes). The trained field staff distributed and 
collected the TCRs while on-site. 

Another aspect of the data collection was the videotaping of each 
preschooler and parent during the three data collection cycles in the two 
implementation years. Trained field staff videotaped the parent and child engaged in a 
book reading activity and in playing with a toy. Both the book and the toy were 
supplied by the field staff, who were trained in videotaping during a week-long 
training and given a script to follow. The videotaping took place at each Even Start 
project, in a setting provided by the Even Start project staff. More details on the 
videotaping can be found later in the chapter (see section entitled Parent Outcomes). 

Finally, each preschooler's hours of participation in preschool education in 
the Even Start project was collected from Even Start project staff using a template 
developed for the CLIO study referred to as the Instructional Services Participation 
Form and described later in the chapter (see section entitled Instructional Outcomes). 
Projects submitted the participation information to the study team monthly. 

Data Collected From Parents 

At each data collection cycle, trained staff administered a battery of one-on- 
one parent assessments as well as a parent interview. The battery covered receptive 
vocabulary, basic reading skills, and comprehension. More detail on the assessments is 
provided later in the chapter (see section entitled Parent Outcomes). The interview 
collected parents' self-reported information about their reading and language activities 
with their child, the home literacy environment, their ratings of their child's behavior 
and social skills, and parent demographics. The study team conducted a week-long 
training prior to each data collection cycle to prepare field staff for data collection. (The 
same field staff conducted the child assessment, the parent assessments, the parent 
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interviews, and the videotaping.) Assessments and interviews were conducted at each 
Even Start project, in a setting provided by the Even Start project staff. 

The videotaping is discussed above under data collected from preschoolers. 

Finally, each parent's hours of participation in parenting education and 
parent-child activities in the Even Start project were collected from Even Start project 
staff using a template developed for the CLIO study referred to as the Instructional 
Services Participation Form and described later in the chapter (see section on 
Instructional Outcomes). Projects submitted the participation information to the study 
team monthly. 

Data Collected From Classrooms 

In the spring of each year, preschool education classes, parenting education 
classes, and parent-child activities classes were observed by trained staff. The study 
team provided extensive training in the use of the observation measures. There were 
two sets of staff for the observations, each trained separately: one set for the preschool 
classes, and one set for the parenting and parent-child classes. The observation 
protocols were designed to collect information on instructional practices in the 
classrooms and on the fidelity of implementation to the CLIO curricula. Observation of 
instructional practices is described in detail later in this chapter (see section on 
Instructional Outcomes), and observation for fidelity of implementation is discussed in 
chapter 4. 



Each spring, teachers were asked to complete a short survey to provide 
information on their educational background, demographics, and professional 
development opportunities. The observers distributed and collected the teacher surveys 
while on-site to conduct the classroom observations. 

Data Collected From Projects 

Each Even Start project director was administered a survey to collect 
information regarding services in each of the four Even Start components, including 
numbers of families and children served, hours offered, and curricula used. Project 
directors were also asked to provide information on their educational background. 
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demographics, and professional development opportunities. The project director survey 
was conducted as a mail survey. 



Sample Sizes 

Children enrolled at CLIO projects were generally eligible for participation 
in the CLIO study if they were between 36 and 60 months of age at the time of 
assessment and were not yet attending kindergarten. 11 Children whose attendance at 
CLIO centers overlapped multiple data collection periods were assessed multiple times, 
but there was no effort to make the sample longitudinal. Sample sizes for analysis 
purposes are shown in figure 2-2 and table 2-8. 



Development of Outcome Measures 

The CLIO study collected a large amount of information in many different 
outcome domains to fully address the study's research questions. There are three broad 
measurement categories: (1) child outcomes, (2) parent outcomes, and (3) instructional 
outcomes. (Table 2-9 shows the outcome measures and the constructs they were 
selected to measure.) Child and parent outcomes align directly with the primary CLIO 
research goals of improving child language and literacy and parenting practices. While 
the instructional outcomes can be viewed as either mediating variables or as outcomes 
in their own right, the principal analysis for this study treated the instructional process 
variables as outcomes. However, secondary analyses were run with non-experimental 
techniques to explore the relationships between instructional process variables and 
child and parent outcomes. 



11 For spring data collection (2004, 2005, and 2006), the child must have turned 3 no later than March 1. 
For fall data collection (2003 and 2004), the child must have turned 3 no later than October 1. In spring 
2006, children who had been assessed in spring 2005 and who were old enough to be in kindergarten 
were not included in the preschool sample, whether or not they were still in preschool. This change was 
made so that these children could participate in the follow-up data collection with the instruments 
designed for kindergarten students. 
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Figure 2-2. CLIO Cross-Sectional Sample Sizes 



Fall D3 
Pre-K 
n=1490 



Spring D4 
Pre-K 
n=1443 




CLIO Intervention starts- 



Fall D4 
Pre-K 
n=13B6 



Spring D5 
Pre-K 
n=1498 




Spring 06 
Pre-K 
n=1292 



NOTE: Figure counts are limited to children who took at least one of the child assessments and were 
enrolled at a project for at least 28 days as of the assessment date. 



Table 2-8. Cross-sectional Child Sample Size by Child Age and Data Collection 
Cycle 



Child age at Baseline CLIO curricula in implementation 



assessment 


Fall 2003 


Spring 2004 


Fall 2004 


Spring 2005 


Spring 2006 


3 


574 


367 


571 


400 


353 


4 


726 


658 


685 


683 


603 


5 


190 


412 


130 


413 


335 


6 


0 


6 


2 


2 


1 


Total 


1,490 


1,443 


1,388 


1,498 


1,292 



NOTE: Counts are limited to children who took at least one of the child assessments and were enrolled 
at a project for at least 28 days as of the assessment date. 
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Table 2-9. CLIO Outcome Measures 





Outcome 


Data collection 
instrument 


Mode of data 
collection 


Domain 


CHILD 


1 


Expressive language: English 


Individual Growth and 
Development Indicator 
(IGDI) 


Child assessment 


Emergent 

literacy 


2 


Expressive language: Spanish 


3 


Receptive vocabulary 


Peabody Picture 
Vocabulary Test (PPVT) 


4 


Phonological awareness: Elision 


Comprehensive Test of 
Phonological and Print 
Processing (Preschool — 
CTOPPP) 


5 


Phonological awareness: Blending 


6 


Print knowledge 


7 


Syntax and grammar 


Test of Language 
Development (TOLD-3) 


8 


Social competence 


Teacher rating form 


Teacher rating 


Socio- 

emotional 

development 


PARENT 


9 


Parent interactive reading skill 


Read Aloud Together 
Profile & Parent 
Interview 


Video observation, 
parent report 


Parenting 

skills 


10 


Parent responsiveness 


11 


Reading & vocabulary skill 


Parent assessment 
battery 


Parent assessment 


Parent 
language & 
literacy 


INSTRUCTIONAL 


12 


Support for oral language 
development 


Observation Measures of 
Language and Literacy 
Instruction (OMLIT) and 
Parenting Education and 
Child/Parent 
Observation (PECAP) 


Classroom 

observation 


ECE 

classroom 

instruction 


13 


Support for print knowledge 


14 


Support for phonological 
awareness 


15 


Support for print motivation 


16 


Literacy resources in classroom 


17 


PE time spent on child literacy 


PE classroom 
instruction 


18 


PE time spent on parenting skills 


19 


PC time spent interacting on child 
literacy activities 


PC classroom 
instruction 


20 


Child: Monthly hours of ECE 
instruction received 


Instructional Services 
Participation Form 
(ISPF) 


Project report 


Participation 

amount 


21 


Parent: Monthly hours of PE and 
PC instruction received 



A key consideration in creating outcome measures was the total number to 
construct. The experimental curricula were expected to possibly affect a broad range of 
child, parent, and instructional outcomes. This argued for a large number of outcome 
measures. However, false positive findings can be caused by running a large number of 
statistical tests. This argued for being parsimonious in selecting outcome measures. A 
target of about 20 outcomes was set so that the expected number of false positive 
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findings in the event of no true effects would not be more than one. Multiple 
comparison adjustments were used for the variety of contrasts of interest pertinent to 
each outcome, but these adjustments were not applied across outcomes, as to do so 
would have lowered statistical power too precipitously. 

Multiple literacy subtests, targeting different aspects of literacy, were used 
for both parents and children. The literacy subtests were averaged together for parents 
but left separate for children. In both cases, there are substantial correlations among the 
tests, but given that the CLIO curricula do not systematically vary adult literacy 
education, parent subtests were averaged to reduce multiple comparison problems. The 
children's subtests were kept separate partly because of the different theory behind 
each subtest and partly to facilitate subsequent meta-analyses since other studies tend 
to report them separately. 

Child Outcomes 

Even Start projects provide ECE to children in low-income families to 
prepare them for success in school. Hence, the CLIO curricula were selected, in large 
part, for the strength of their preschool curricula, in particular the language and literacy 
dimensions. The CLIO child assessment battery was designed to measure early 
language and literacy development, including vocabulary, phonological awareness, and 
print knowledge. Research has shown these areas are important in the development of 
reading skills and predictive of school achievement (National Research Council 2001). 

On each of the child outcome tests (except the IGDI and Spanish IGDI), 
multiple scoring procedures were applied, including a simple count of items correct as 
well as complex scoring similar to what is done in the National Assessment of 
Educational Progress (NAEP), the Early Childhood Longitudinal Study - Birth Cohort 
(ECLS-B), and several other large-scale testing programs. (Appendix B provides 
descriptions of the two scoring procedures.) Where the complex scoring was carried 
out, the scores were scaled to have a mean of 250 and a standard deviation of 50. The 
raw scores were left on the metric represented by the number of items asked. For all 
contrasts between study groups, treatment effects are expressed in terms of standard 
deviations within the control group. 
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Below we describe each of the outcome measures. See appendix B for 
psychometric data for each test. 

Expressive Language. The IGDI Picture Naming subtest (Early Childhood 
Research Institute on Measuring Growth and Development 2003) measures expressive 
language by asking the child to recognize and name a series of common objects in 
English using picture cards. Following the test publisher's standard procedure, the 
assessor counted the number of cards the child named correctly in 1 minute, but did not 
track which cards the child failed to name correctly. The protocol called for the subtest 
to be administered to all children regardless of native language. A parallel subtest, the 
Spanish IGDI, was administered in Spanish to children from Spanish-speaking families. 

Receptive Vocabulary. The Peabody Picture Vocabulary Test (PPVT-III) 
measures receptive vocabulary (Dunn and Dunn 1997) and has been widely used in 
other early childhood studies. In this subtest, the child demonstrates his or her 
understanding of the meaning of an English word by pointing to the correct picture. 
CLIO used a version of this test that was adapted by Westat for this study. This adapted 
version contained fewer items, and the words were divided into three sets. In the first 
set for every child, there were 14 words. If the child made fewer than three errors in the 
first set, then he/she was given an additional set (the ceiling set) of 10 more difficult 
words. At the other extreme, if the child made more than seven errors in the first set, 
then he/she was given an additional set (the basal set) of eight easier words. 

Phonological Awareness: Elision. The Preschool CTOPPP (Lonigan, Wagner, 
Torgesen, and Rashotte 2002) was developed to measure phonological awareness in 
English in younger children. The Elision subtest measures the child's ability to 
recognize English word parts, such as components of compound words, syllables, and 
phonemes. The examiner reads a compound word and the child is asked to identify 
what is left when part of the word is taken away. The part taken away can be a whole 
word from a compound word (e.g., "toothbrush without brush") or phoneme (e.g., 
team without /m/). There were 18 items in this subtest. For the first nine items, the child 
identified what was left by pointing at a picture of it on a page with four pictures. For 
the last nine items, the child was asked to verbalize the remainder without visual aids. 
The first nine items were given to each child regardless of the child's error rate. After 
the first nine items, a run of three consecutive errors caused the subtest to be stopped. 
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Phonological Awareness: Blending. The blending subtest of the Preschool 
CTOPPP measures the child's ability to combine English word parts, such as 
components of compound words, syllables, and phonemes. The examiner says two 
parts of an English word and asks the child to put them together (e.g., "horse and shoe 
together is horseshoe"). There were 21 items in this subtest. In the first nine items, the 
child was asked to identify compound words by pointing at the answer from a page 
with four choices. The child went through all nine items regardless of the number of 
errors. For the final 12 items, the picture support was not available, and there was a 
skip-out rule tied to three consecutive errors. 

Print Knowledge. The Print Awareness subtest of the Preschool CTOPPP 
assesses the child's ability to identify Roman alphabet symbols that represent letters 
and words used in English, to identify specific letters by name and by sound, and to 
produce letter sounds. On this subtest, there was no stopping rule. All children were 
asked all items. 

CLIO used a research version of the Preschool CTOPPP available in the 
study's first data collection year (2003-2004). However, a slightly revised version of the 
test with normed scores has since been published by ProEd as the Test of Preschool 
Early Literacy (TOPEL). The TOPEL Phonological Awareness test combines the 
CTOPPP Elision and Blending subtests, contains fewer items, and uses different 
stopping rules. The TOPEL Print Knowledge test contains the same items as the 
CTOPPP Print Awareness subtest, but administers the items in a different order with 
different administration rules. 

Syntax and Grammar. The TOLD-3 Grammatic Understanding subtest 
(Newcomer and Hammill 1997a; 1997b) measures the child's ability to comprehend the 
meaning of an English sentence, with an emphasis on syntax and morphology. 
Knowledge of syntax is important in constructing and understanding sentences. In this 
subtest, the assessor read a sentence aloud, and the child was asked to select one picture 
from three possible choices that correctly corresponded to the sentence. There were 24 
items in this subtest. Six consecutive errors caused the subtest to be stopped. 

Child Social Competence. Developing children's social competence is an 
important objective for early childhood programs, and the development of social skills 
and positive behaviors is associated with success in school. Positive behavior includes 
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cooperation with adults, friendly play, and sharing with other children. Problem 
behaviors include disruptive or overly aggressive behavior, hyperactivity, excessive 
shyness and social withdrawal. These negative behaviors are associated with problems 
in school and/or receipt of psychological help (Gresham and Elliott 1990). 

A social competence scale was created from preschool teacher reports on the 
behavior and social skills of children in the study. The scale combines information from 
two different sets of items in the CLIO Teacher's Rating Form — cooperative behavior 
and problem behavior. (Appendix C contains details on the construction of the child 
social competence scale.) 

Parent Outcomes 

Two types of parent outcomes were of interest: improved parenting skills 
and improved parent literacy. The hypothesis was that both help parents to be their 
child's first teacher. As with instructional outcomes, parent outcomes were analyzed 
both as outcomes and as mediators for child outcomes. 

Parenting Skills. The parenting curricula implemented by CLIO projects 
focused on showing parents how to be effective teachers of their child and emphasized 
teaching early reading skills. It has been shown (Whitehurst and Lonigan 1998) that the 
practice of specific behaviors during joint book reading can promote children's 
engagement in reading and help them better comprehend the story and understand the 
conventions of print. Mutual questioning and responding, making stories relevant to 
the child's life, giving praise and feedback, explaining, physically sharing the book, 
monitoring a child's understanding, and adjusting language are all behaviors that 
enhance children's literacy skills and comprehension. Given the second primary 
research question of determining the added value of a parenting curriculum with a 
focus on child literacy, we developed the instruments for measuring parenting behavior 
with a particular focus on those aspects of parenting that theory suggested should 
promote child literacy. 

Parenting skills were measured by coding videotaped parent-child 
interactions and by parent self-report. Both of these measured parenting behaviors that 
were thought to be important in differentiating parents who were more or less effective 
teachers (See appendix D for more information on the coding of the interactions.) A 
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total of 90 variables describing parenting behaviors were measured in the spring of 2005 
and 2006, so some distillation was required. Instead of sorting the variables a priori into 
groups based on the literature of the field, the data on these variables were empirically 
combined into two outcome scales: 

• Parent interactive reading skill, and 

• Parent general responsiveness to the child. 

The procedures used in this process included variable clustering and factor 
analysis within clusters. These procedures result in an unequal weighting of the items 
assigned to each scale. The scale for parent interactive reading skill has 49 items, while 
the scale for parent general responsiveness has 41. The correlation between the two 
scales is 0.6. 

Despite the lack of a priori grouping, these two scales align fairly well with 
two primary dimensions of teacher quality at the pre-k and elementary level recently 
identified by Hamre and Pianta (2005): instructional support and emotional support. 
(See appendix D for details on how these scales were created.) They also align well with 
the goals set for parents by the CLIO parenting curricula. 

There is no prior information on the reliability or validity of these scales. 
Some of the relationships explored in chapter 7 support the validity of the second scale 
in terms of its relationship with the targeted child outcomes. In appendix D, we provide 
information on the training of the coders and the ways in which we established rater 
reliability. 



Parent Language and Literacy. Improving parent literacy is one of Even 
Start's main goals. While AE was not varied as part of the CLIO curricula, parent 
language and literacy have a strong relationship with child outcomes. Further, parent 
literacy may be positively affected by participation in parenting education with a child 
literacy focus. 

The CLIO parent assessment was designed to measure English language and 
literacy outcomes, including vocabulary, basic reading, phonics, and comprehension. A 
single outcome measure was created from four tests: the PPVT and three Woodcock- 
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Johnson subtests (Letter-Word Identification, Passage Comprehension, and Word 
Attack). Using spring 2004 data (first plausible value of each IRT score), Cronbach's 
alpha for the scale was 0.95 (see appendix B for information about scoring procedures 
and psychometric properties for each of these subtests). 

The Peabody Picture Vocabulary Test (PPVT-III) (Dunn and Dunn 1997) 
measures receptive vocabulary. In this test, the parent demonstrates his or her 
understanding of the meaning of a word by pointing to the correct picture after the test 
administrator reads a test word aloud. 

Three Woodcock-Johnson subtests were administered (Woodcock, McGrew, 
and Mather 2001). Letter-Word Identification measures basic reading skills and requires 
respondents to identify printed letters and words with an oral response. There are 76 
items in this subtest. Word Attack measures the subject’s skill in applying phonic and 
structural analysis skills to the pronunciation of unfamiliar printed words. The subject 
reads aloud letter combinations that are linguistically logical but that form nonsense 
words or low-frequency words in English. There are 32 items in this subtest. Passage 
Comprehension measures comprehension and vocabulary skills. In the first several 
items, respondents point to the picture represented by a phrase. The remaining items 
require reading a short passage and supplying an appropriate answer for a missing key 
word. There are 47 items in this subtest. The test publisher's skip-out rule of stopping 
after six errors in a row was used for all three subtests. 

Instructional Outcomes 

If the curricula work as designed, significant changes in instructional 
practices were expected to occur. These changes were hypothesized to be necessary (if 
not sufficient) for impacts on children and parents. Knowing whether such changes 
occurred would be useful in interpreting the level of impacts on children and parents. 
Moreover, an understanding of how instruction changed in treatment projects is 
important to designers of future curricula. Although there was not a strong hypothesis 
about whether introducing research-based, literacy focused curricula would increase 
participation levels, level of participation was measured, since it was also a possible 
mediator of impacts on children and parents. Accordingly, systems were developed to 
measure instructional practices and participation. 
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Instructional practices were evaluated based on direct observation by 
experienced education researchers using standardized schedules, as is described below 
in more detail. As with parenting, many variables were created from these 
observational measurement systems, and so some distillation was required. From these 
detailed measurements, we developed eight instructional outcomes. Of these, five focus 
on ECE, two on PE, one on PC. In addition, based on monthly information from 
projects, we developed two participation measures — one for children's participation in 
ECE classes and one for parent participation in PE and PC classes. The alignment of 
particular measurements with the scales was straightforward because the intended 
scales guided the development of the measurement systems. Flowever, not all the 
measurement elements worked as intended; some of these were dropped from the 
scales as discussed in appendices E and F to improve the reliability of the scales. 

Preschool Instruction. The CLIO preschool curricula were intended to 
promote aspects of language development and emergent literacy skills that have been 
shown to be essential to proficient reading. As described by Whitehurst and Lonigan 
(1998), the elements of emergent literacy form two domains: (1) inside-out processes, 
which are rule-driven processes for rendering the written symbols of text into sound 
and vice versa (i.e., decoding and encoding), and (2) outside-in processes, which are 
sources of information outside the printed text, such as vocabulary, background 
knowledge, and contextual knowledge, that support understanding (and decoding) the 
text. Other terms used to describe emergent literacy are oral language and print 
motivation (outside-in), and phonological processing/sensitivity and print knowledge 
(inside-out). These terms are defined as follows: 

• oral language: lexical/conceptual, semantic, and syntactic abilities, 

• print motivation: interest in reading and writing activities, 

• phonological processing: sensitivity to and ability to manipulate word 
sounds, and 

• print knowledge: knowledge of units of print (letters, words), ability to 
translate print to sound and sound to print (letter-sound, and ultimately 
word-sound). 

The primary mechanism hypothesized to improve developmental outcomes 
for Even Start children in CLIO is a preschool curriculum that focuses on teaching these 
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skills. The two preschool curricula were selected for CLIO, in part, because they include 
instructional activities in all four areas of emergent literacy, albeit differing in relative 
attention across areas. 

To assess whether the curricula were successful in changing teaching 
activities in the critical areas of language and literacy, a classroom observation measure 
was developed specifically for the CLIO study: the Observation Measures of Language 
and Literacy Instruction, or OMLIT (Goodson, Layzer, Smith, and Rimdzius 2004, 2006). 
The OMLIT is a battery of six measures that focus on aspects of classroom practice that 
have been shown in research to support children's language development and 
acquisition of early literacy skills. Appendix E describes the development of and 
rationale for the OMLIT and provides psychometric information on the battery. There is 
no prior information on the reliability or validity of these measures. 

Classroom observations using the OMLIT were conducted each spring in 
CLIO classrooms Along with the ECE OMLIT, observers also completed the Arnett 
Rating of Caregiver Behavior for each lead teacher (Arnett 1989). The Arnett rated the 
teacher's engagement with, responsiveness to, and affect toward children in the 
classroom. In the spring 2004 baseline data collection only, observers also completed the 
ECERS-R, which rates overall classroom quality in six classroom areas. (See earlier 
discussion about the ECERS-R under classroom instruction, pages 22-23.) 

Five outcome constructs were derived from the six ECE OMLIT measures to 
correspond to key elements of preschool instruction that are being manipulated by the 
curricula. (Appendix E provides details on how the preschool instructional outcome 
measures were constructed.) These included constructs for the four components of 
emergent literacy, and an additional construct — the adequacy of language and literacy 
resources in the classroom — which is commonly considered to be related to children's 
emergent literacy (although no strong research evidence exists to support this claim). 
The five instructional outcome variables are the extent to which the preschool classroom 
provides: 



• support for oral language, 

• support for phonological awareness. 
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• support for print knowledge, 

• support for print motivation, and 

• adequacy of literacy resources in the classroom. 

PE Instruction. Even Start requires that each project provide PE instruction, 
designed to increase parents' knowledge about early childhood development and 
parenting behaviors and practices, toward the objective of helping parents contribute 
actively and constructively to the literacy development and school readiness of their 
children. Prior research has shown that Even Start projects often use PE to provide 
instruction in many areas that are not directly related to child literacy or how to 
effectively interact with their children. Examples of topics in other areas include adult 
life skills, household management, health, and nutrition (St.Pierre et al. 2003). Given 
that the core strategy for the CLIO parenting curricula was to intensify the focus on 
child literacy, the study team decided to measure how parenting education time was 
allocated across these three broad areas: how parents can directly promote child 
literacy, how parents can interact more effectively with their children (abbreviated as 
"parenting skills"), and other topics. 

An observational measure called the Parenting Education and Child and 
Parent Observation (PECAP) was created specifically for the CLIO study to measure the 
amount and type of activities being undertaken both in PE and PC classes. Appendix F 
provides details on the development of and psychometric information for the PECAP. 
There is no prior information on the reliability or validity of these measures. 

Based on observations made each spring with the PECAP, two outcomes 
were created for PE classes: 12 

• the percentage of PE class time spent on child literacy activities: 
reading/looking at books/letters; writing/emergent writing; oral 
language, songs, rhymes, sound games; and 



12 The PECAP also records time spent on adult-focused activities, such as parent health and on other 
activities such as play activities, but these variables were not used in the CLIO impact analysis. 
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• the percentage of PE class time spent on parenting skills: responding to 
and managing child behavior; home-school relations; ideas for home 
play; child development; child health, well-being, safety. 

Most important, from the perspective of this evaluation, is the percentage of 
PE class time spent on child literacy, since that variable is closely aligned with the 
central thrust of the CLIO parenting curricula, as well as with Even Start's broader 
guidance (U.S. Department of Education 2003) that PE should be directed at skills that 
allow the parent to be the "primary teachers for their children." The percentage of PE 
class time spent on parenting skills also is important because these activities may, in the 
long run, contribute to a parent's ability to be a good teacher for his/her children. The 
PECAP was revised between spring 2004 and spring 2005, and there were considerable 
missing data in spring 2004, so no baseline data were available for this measure. 

PC Literacy Activities. The legislation authorizing Even Start requires that 
projects provide interactive literacy activities for parents and their children (U.S. 
Department of Education 2003). These activities may take place in preschool 
classrooms, as part of PE classes, in separate PC classes, or during home visits. As with 
PE, prior Even Start evaluations have shown that many Even Start projects use PC time 
for a wide assortment of non-literacy activities (St.Pierre et al. 2003). 

The PECAP observational measure (described above in the discussion of PE 
outcome variables) was used to describe the amount and type of PC literacy activities in 
CLIO projects. Unlike PE, where the focus is on child literacy activities in which parents 
are being shown how to be their child's teacher, in PC literacy activities the focus is on 
literacy activities where the parent gets the opportunity to practice what he/she learned 
in PE by interacting with his/her child. The PECAP records whether each activity 
involves (1) parents only, (2) children only, (3) both parents and children, with 
parent/child pairs interacting together, or (4) parents and children but without any 
interaction in parent/child pairs. The outcome variable created from the PECAP data 
was 



• the percentage of PC time in which parents and children were 
interacting on activities that were directly related to child literacy. 
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As with PE outcomes, changes in the PECAP from spring 2004 to spring 2005 and 
missing data in spring 2004 mean that no baseline measurement was available for this 
measure. 



Extent of Participation in Even Start. It was hypothesized that the amount 
of Even Start instructional services received by CLIO parents and children might vary 
by study group. Perhaps families in the experimental groups were more engaged by the 
CLIO curricula and therefore missed fewer days of Even Start than families in the 
control group. The Instructional Services Participation Form (ISPF) was used to collect 
participation information from CLIO projects monthly. CLIO project staff were asked to 
report the number of hours of instruction in which each child and parent participated in 
each of Even Start's four instructional components (ECE, PE, PC, AE). Projects 
submitted data via an on-line ISPF data collection system, email, and fax. Editing and 
follow-up were performed on an on-going basis. Two outcome variables were 
constructed from the ISPF data: 

• the number of hours per month that a child participated in ECE, and 

• the number of hours per month that a parent participated in PE and PC. 

These two scales were created by counting the hours that a child or parent 
participated in Even Start across a 9-month period (September through May) 13 and then 
dividing the total by nine — regardless of whether the participation was all within a 
single month or spread more evenly across months. 14 Children enrolled for only brief 
periods have low participation scores, as do children enrolled for longer periods but 
with rare attendance. Children with high weekly attendance over a long enrollment 
period have high participation scores. 



13 Although Even Start is intended to be a year-round program, projects may either not provide services 
in the summer, or the services may differ greatly from those provided during the school year. Reports 
from the summer months were particularly ambiguous. To reduce the impact of this ambiguity on the 
participation outcomes, participation was counted only from September through May of each year. 

14 We also have a 7-month version of each participation measure. The 7-month versions average 
participation hours from September through March. The 7-month versions were used as putative causal 
agents in some analyses in chapter 7 and thus need to avoid containing data about the period following 
the literacy assessments. Since chapter 3 is about the process that could have affected results, the 7- 
month version is also used there. The 9-month versions were used as intervention outcomes in chapters 
5 and 6. 
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3 . 



CURRICULUM IMPLEMENTATION 



This chapter discusses the implementation of the CLIO curricula. Our 
examination of CLIO curriculum implementation includes the following components: 

• description of the curricula, 

— What was the intended curriculum — that is, the desired teaching 
practices in the preschool and parenting education classes? 

— What types of training and support were provided to project staff to 
help support high-fidelity implementation of each curriculum? 

• fidelity of implementation of the curricula, 

— How was fidelity measured for each curriculum? 

— To what extent were the curricula implemented as planned? 

• level of exposure of children and parents to the curricula, and 

• types of instruction provided in the control group. 

Answering questions about fidelity of implementation requires a definition 
of what constitutes the critical components of each curriculum (content and process) 
and a method for measuring the match between the component as intended and the 
component as implemented. These activities pose challenges for any study but 
especially for CLIO, where multiple curricula were implemented across a large and 
heterogeneous sample of classrooms. 



Curricula as Intended 

The following descriptions of the curricula expand on the descriptions in 

chapter 2. 
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Let's Begin with the Letter People plus Play and Learning Strategies 



This combined curriculum consists of the Let's Begin with the Letter People 
preschool curriculum described below plus the Play and Learning Strategies parenting 
curriculum. 

Let's Begin with the Letter People-Preschool Curriculum. As designed for 
the CLIO study, this curriculum included Let's Begin with the Letter People (Abrams 
and Company Publishers), a published early childhood education classroom 
curriculum, augmented by CIRCLE teacher training on developmentally appropriate 
techniques for promoting language and literacy skills in preschool (CIRCLE is the 
Center for Improving the Readiness of Children for Learning and Education, the state 
center for early childhood development at the University of Texas, Houston Health 
Sciences Center). 

Let's Begin with the Letter People. This literacy program for pre-kindergarten 
children adheres to the research findings that the most effective instruction for 
preparing children to become lifelong readers and learners builds progressively on 
children's understanding and use of both spoken and written language, specifically 
focusing on the four critical domains of early literacy: 

1. Oral language, oral comprehension, and vocabulary; 

2. Phonological and phonemic awareness; 

3. Letter recognition; and 

4. Conventions of print. 

The unique feature of Let's Begin with the Letter People is that it is built 
around 26 imaginary characters that represent the letters of the alphabet. The 
curriculum uses these characters to help children learn about letters, sounds, and 
concepts. 15 The curriculum is organized into five themes: All About Me; Getting Along 
With Others; Everyone Has Needs; Animals, Animals, Animals; and Nature All Around Us. 
Each resource book contains five or six lessons, each dealing with a different facet of the 



15 Letter People are inflatable characters, representing the letters of the alphabet. They wear a capital letter 
on their front and a lower case letter on their back to support letter identification. Each has a 
distinguishing characteristic that is readily associated with the sound of their letter (e.g., Mr. N has a 
noisy nose and likes to eat noodles) 



42 




theme, and is introduced with the help of a different Letter Person. For example, the All 
About Me theme contains the following six units: 

1. What's My Name? (Mr. N). 

2. What Will I Wear? (Ms. W). 

3. My Body (Ms. P). 

4. I'm Healthy (Mr. H). 

5. All About My Senses (Mr. M). 

6. We All Have Feelings (Ms. A). 

Each unit, in turn, contains six lessons. Each lesson plan provides a daily 
menu of activities — beginning with a brief whole-class Meeting Circle activity, followed 
by a selection of individual and small-group Interest Center activities. 

As an example, the first unit in the All About Me theme is called 'What's My 
Name?' Mr. N introduces this unit, which focuses on the letter N/n, and the /n/ sound, 
and introduces relevant vocabulary words. In the Meeting Circle Time, the storybook to 
be read to the class is 'From Ann to Zach', and the read-along book is Mr. N's 'Wlmt’s My 
Name ?' An example of an activity in an Interest Center is making placemats 
individualized with each child's name in the art center. 

Each Even Start classroom assigned to Let's Begin with the Letter People was 
provided with the following set of materials: 

• Letter People and their accessories, 

• Letter People finger puppets, 

• Big and Little books and their story tapes, 

• Musical materials, 

• Family materials, 

• Manipulative letters, numbers, and shapes, 

• Me Bags, 16 and 

• Teacher resource materials. 



16 Me Bags are bags that children take home and fill with meaningful objects, which they then explain to 
their classmates. Each Letter Person also has a Me Bag that is shared with the class to reinforce concepts 
being taught. 
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For additional information about Let's Begin with the Letter People, see 
http://www.abramsandcompany.com/lets begin with letter people.aspx . 

CIRCLE Training. In conjunction with the Let's Begin with the Letter People 
curriculum, CIRCLE provided professional development to the early childhood 
education staff on providing developmentally appropriate literacy instruction. This 
training covered the following topics: 

• rich language input, 

• responsiveness to children's signals, 

• maintaining and building on interests, 

• choice-providing strategies, 

• monitoring of children's behavior, 

• language development, 

• print and book awareness, 

• motivation to read/read aloud, 

• phonological awareness, 

• letter knowledge and early word recognition, and 

• written expression. 

Play and Learning Strategies (PALS) -Parenting Curriculum. For the CLIO 
study, the Let's Begin with the Letter People curriculum was combined with the PALS 
parenting education program developed by the University of Texas-FIouston Flealth 
Sciences Center. PALS also included a certification requirement for parents. 

PALS is designed to teach parents responsive parenting strategies that can 
help their children build the cognitive and language skills they will need for success in 
school. PALS has previously been evaluated with parents whose children are at risk for 
developmental delay and academic failure due to poverty, low family literacy, and 
other risk factors. Parents who participate in the PALS curriculum learn two categories 
of behaviors: 

1. behaviors that provide a supportive, dynamic interpersonal 
environment for learning, such as attending to children's 
communicative signals, responding promptly and warmly, and 
following children's interests to encourage further interactions with 
learning materials and people; and 
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2. behaviors that directly help children build their cognitive, language, 
and pre-literacy skills, such as teaching new words, using rich language 
to describe and connect objects and actions, building on children's 
utterances to model longer sentences, and reading books with children 
in ways that encourage their active participation. 

To help parents learn these behaviors PALS instructors focus on the 
following four key concepts: 

1. The zone of proximal development: this is the learning level at which adult 
support is critical for helping children accomplish goals beyond what 
they can do by themselves. 

2. Scaffolding: when an adult builds upon what the child can do and helps 
him/her take the next step by expanding on what the child has said or 
done. As the child experiences this support consistently over time, 
greater cognitive and social skill development is expected to occur. 

3. Warm, responsive style: children are not likely to sustain their 
participation in learning activities if parents ignore them or respond 
harshly to them. In contrast, when parents respond with warmth and 
encouragement to their children's efforts, children are more likely to 
stay engaged and attentive, thereby allowing them to learn much more 
from each interaction. 

4. Spiral learning: children need repeated exposure to new concepts and 
behaviors over time, with multiple opportunities to practice and 
integrate their new knowledge into their existing skill set. Mastery 
happens gradually as children build a network of connections between 
new and old skills. 

Parents learn these concepts so that they can be better teachers of their own 
children. In addition, parent educators are taught to rely on these concepts while 
working with Even Start parents. That is, during a parent education session, the PALS 
instructor will scaffold parent knowledge, use a warm and responsive instruction style, 
and will spiral back to previously learned topics. 

Each PALS center-based session uses the following format: a review of the 
previous week's topic and discussion of the parents' use of that skill, introduction of a 
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new skill though discussion and an instructional videotape, practice of the new skill 
with their child while the parent educator videotapes and provides coaching, and 
discussion as the group and parent educator review the videotape together. 

In addition to these center-based parenting education sessions, PALS also 
provides activities for home visits. Home visitors review and reinforce concepts that 
parents may be having difficulty implementing. Further, the home visitor provides an 
important link between the preschool classroom themes from Let's Begin with the 
Letter People and the PALS concepts, which are reinforced through the home activities. 
Materials provided to the parenting educator by the PALS program included the 
following. 



• facilitator manuals in English and Spanish, 

• video manuals in English and Spanish, 

• home visitor manuals in English and Spanish, 

• videotapes in English and Spanish, 

• coaching toy bag, 

• home visitor toy bag, and 

• parent workbooks. 

Certification. As part of the CLIO study, each Even Start parent educator who 
was implementing PALS was required to become certified as a PALS facilitator. The 
certification process included the requirement that the participant submit up to three 
videotapes of themselves leading PALS instructional sessions. The first of these tapes 
included the facilitator conducting a complete mock session (introduction, video, and 
summary, transition, filming, coaching, and reviewing). The second tape was also a 
complete session but this time with parents. A third tape was requested if the facilitator 
needed further follow-up suggestions. The tapes were reviewed and appropriate 
feedback provided to each facilitator-in- training before award certification was granted. 

For further information about PALS, see 
http://www.childrenslearninginstitute.org/our-programs/program- 
overview/PALS/default.html . 
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Partners for Literacy 



Although some aspects of the Partners for Literacy curriculum existed prior 
to CLIO, 17 the current version that includes both a preschool and a parenting 
curriculum was developed specifically for the CLIO study. The developers further 
refined some of the materials and strategies over the course of the 2 implementation 
years. Partners for Literacy was developed by Barbara Wasik and Joe Sparling from the 
Frank Porter Graham Child Development Center at the University of North Carolina, 
Chapel Hill. 

Partners for Literacy — Preschool Curriculum. (Wasik and Sparling, 
University of North Carolina, Frank Porter Graham Child Development Center). This 
early childhood education curriculum focuses on language and literacy activities for 
children, coupled with specific instructional strategies for teachers. It encompasses four 
key early literacy domains: oral language and vocabulary, letter knowledge, 
phonological awareness, and concepts of print, as well as social and emotional 
development. Educators in a Partners preschool classroom are to create a literacy-rich 
classroom, promote language and literacy skills throughout the day, and provide both 
group and individualized instruction. Instruction is individualized by interacting with 
successive pairs of children throughout the day on both game-like activities and book 
reading. The activities and instructional strategies are described below. 

LiteracyGames is one essential part of the curriculum. LiteracyGames is a 
series of 50 instructional activities (games) for 3-year-olds and 50 games for 4-year-olds, 
designed to be played by one adult and two children. The games address such early 
literacy domains as concepts of print, letter knowledge, oral language, phonological 
awareness, writing, and creativity. Each has a specific instructional goal and takes about 
5 minutes. The games are engaging for children and involve considerable back-and- 
forth between adult and child. Children progress from easy to more difficult activities 
within each game. 

Teachers focus on one or two games per week, depending on the length of 
the school day. Each game has three "cycles" allowing teachers to tailor the activities to 



17 For example, some components of Partners for Literacy were used in the now-famous Abecedarian 
study out of the Frank Porter Graham Center. 
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the abilities of individual children. Teachers are expected to set aside a specific time of 
the day in which to play LiteracyGames. For example, during the LiteracyGames time 
the teacher would play a game with successive pairs of children while the rest of the 
class engaged in free play or center time supervised by the teacher assistant. Child 
progress on LiteracyGames is monitored using a Record of Mastery form. 

Interactive Book Reading is another essential part of the Partners for Literacy 
preschool curriculum. Teachers are expected to read for 5 to 10 minutes with each child, 
every day. The teacher may choose to read with children individually or in pairs. The 
curriculum provides a set of children's books to complement those in the classroom and 
encourages teachers to create a lending library of books that children can take home. 
Two other types of books were designed to facilitate Interactive Book Reading: 
Conversation Books and Little Conversation Books. 

Conversation Books are very simple books designed to stimulate 
conversation between an adult and child. They also serve to familiarize teachers with a 
specific interactive book reading strategy known as the 3S Strategy: See, Show, and Say 
(discussed in more detail below). Little Conversation Books are a collection of small 
books designed to build early literacy skills, specifically oral language, phonological 
awareness, print awareness, and alphabet knowledge. The sets include: ABC stories. 
Nursery Rhymes and Poems, Words in Words, Matching Stories, Sequencing Stories, 
and Problem Solving Stories. 

Scaffolding new competencies is the foundation for all Partners for Literacy 
instructional strategies. That is, building on existing knowledge and skills and adding 
support to enable a child to move to a higher level of competence. Scaffolding can take 
many forms, including modeling, leading questions, prompting, and instructional 
conversations. 

The 3N Strategy is a means of structuring interactions with children. It 
consists of three parts: Notice what a child is doing or is ready to do; Nudge or prompt, 
encourage, assist the child to take a step further; and Narrate or give praise, feedback, 
and acknowledgment. This strategy is used throughout the Partners for Literacy 
classroom day. 
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Interactive Book Reading Strategies. These strategies are all designed to 
facilitate an instructional conversation between the child and teacher. 

• 3S Strategy: See, Show, Say. See, Show, and Say refer to the different 
levels of response required of a child during any Interactive Book 
Reading session. A child is first asked to "see" or look at a specific 
feature of the book, such as a picture. If the child follows this direction, 
the teacher can then ask the child to "show" an object or word on the 
page. If the child can follow this direction, the teacher then asks the 
child to "say" a word or answer a question. This strategy helps develop 
active engagement on the part of the child. 

• Wh Questions: Who, What, When, Wliere, Winy. The teacher uses Wh 
questions when a child has even a minimum level of oral language. Wh 
questions are used in a progressively more complex manner by the 
teacher, always individualizing for the child. These questions facilitate 
an ongoing instructional conversation between teacher and child and 
help the child develop comprehension skills. 

• Expanded Book Reading is used to increase motivation to read and 
promote comprehension. It includes a set of activities to facilitate 
literacy skills and comprehension such as discussing a book before or 
after it has been read or incorporating the book into other classroom 
activities such as art or music. 

Enriched Caregiving. Enriched Caregiving, or teaching throughout the day, is 
intended to promote language, literacy, cognitive, and social development. Teachers 
can use Enriched Caregiving strategies to turn everyday routines, activities, and 
transitions into important learning opportunities. For example, a teacher could use 
lunchtime conversation to develop oral language skills or to reinforce a previous lesson 
that dealt with a particular letter sound. 

Problem-Solving Strategies. Problem-solving strategies are used in many ways 
throughout the Partners for Literacy curriculum. These strategies are used to help 
children develop social skills and learn how to manage their own emotions. Teachers 
help children learn to recognize feelings, identify wants and needs, and develop simple 
problem-solving skills at age appropriate levels. 
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A typical day in a Partners for Literacy classroom would incorporate the 
types of activities common in many preschool classrooms such as circle time, free play, 
center time, and structured small group activities in addition to gross motor play, 
meals, etc. The curriculum calls for setting aside time for Partners for Literacy-specific 
activities such as LiteracyGames and interactive book reading — an hour per day for 
half-day programs and 2 hours per day for full-day programs. 

Partners for Literacy — Parenting Curriculum. Partners for Literacy is a fully 
integrated curriculum, so that the parent component parallels the themes and activities 
in the preschool classroom. Parents are taught to use many of the same materials and 
instructional strategies with their children as are used in preschool classrooms. 

LearningGames. LeamingGames are short game-like activities designed for 
parents and children ages 3 to 5 that are similar to the LiteracyGames used in the 
preschool classrooms. LearningGames cover the same early literacy domains and are 
played in the same back and forth fashion as LiteracyGames. They are designed to 
complement the activities in preschool classrooms and give parents the opportunity to 
reinforce these activities at home. 

Interactive Book Reading. The parenting Interactive Book Reading component 
of Partners for Literacy helps parents develop strategies to have a conversation with 
their children when reading a book together. These strategies promote early literacy 
skills. A variety of children's storybooks are used by parents, as well as the 
Conversation Books and Little Conversation Books described earlier. 

Instructional Strategies. Parents are taught the same instructional strategies as 
those used in preschool classrooms, including the 3N Strategy, 3S Strategy, Wh Questions, 
and Expanded Book Reading. Parents are also taught to use Enriched Caregiving 
strategies with their children. The curriculum also helps parents learn a problem- 
solving strategy that includes setting goals and defining possible solutions and their 
consequences. This strategy helps parents address everyday parenting concerns. 

In a parenting session, parent educators use a specific protocol as the 
structure for each lesson: Read, Role Play, and Reflect. As each lesson or strategy is 
introduced, parents read and discuss the information together. They then role play and 
practice a specific strategy with a partner. Finally, parents work as a group to review 
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and reflect on their role play, as they prepare to work with their children during parent- 
child interactive sessions. 



Parent-child interactive sessions provide parents with the opportunity to 
play LearningGames and read a book with their children under the guidance of 
instructional staff. Each interactive session is reviewed in the parenting session prior to 
the introduction of any new activity or concept. 

Complete information about the Partners for Literacy preschool and 
parenting curricula can be found at www.fpg.unc.edu/~literacy. 



CLIO Curricula Professional Development 

As discussed in chapter 2, implementation of the curricula in the field was 
supported by the curriculum developers in three ways: annual group training, on-site 
mentoring, and ongoing support. For the CLIO study, this professional development 
was an integral part of the curricular interventions. 

Training. Each developer conducted 4-day training sessions for project staff 
in the four study groups in the summer of 2004, before the first implementation year. 
The developers provided make-up trainings in the fall of 2004 for staff who were not 
able to attend the summer session. Developers again conducted training sessions 
during the summer of 2005. If project staff were not able to attend the summer or make- 
up sessions, developer staff often provided on-site training during mentoring visits. The 
main challenges were teacher turnover and the widely dispersed treatment projects. 
However, at least one staff member from all projects received training, and most 
teachers (more than 80 percent of early childhood education and parenting education 
staff) reported receiving training on the curricula. 

The 4-day summer sessions included parallel tracks for preschool and 
parenting staff. The training consisted of extensive hands-on practice and small-group 
discussion in addition to large-group overviews and introductions of specific topics and 
teaching strategies. The Let's Begin training included sessions on the five themes and 
associated lesson plans, use of the huggables and other Letter People materials, and best 
practices related to developmentally appropriate literacy instruction. The Partners for 
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Literacy training included sessions on LiteracyGames, interactive book reading, 
enriched caregiving, and English language learners. In addition both developers had 
sessions devoted to classroom organization and management. Project staff received 
detailed manuals describing all aspects of each curriculum. 

Training for parenting staff also included ample opportunity for hands-on 
practice and role-playing. PALS training focused on the concepts and behaviors to be 
taught to parents, reviewed the video lessons, and allowed staff to practice coaching 
parents. After training, teachers produced a videotape for developer staff to become 
certified. Partners for Literacy included sessions on teaching parents about 
LearningGames and book-reading strategies, enriching caregiving strategies at home, 
and coaching parents during parent-child activities. Parenting staff also received 
detailed curriculum manuals. 

On-site mentoring and ongoing support. On-site mentoring consisted of 
visits to each treatment project from developer staff members. During these 2- or 3-day 
visits, developers observed project staff using the curriculum, provided feedback and 
coaching, and supported curriculum planning. On average, each treatment project 
received two mentoring visits per year. 

Curriculum developers also provided CLIO projects with support 
throughout the implementation years. Let's Begin and PALS held regional monthly 
conference calls to allow project staff to discuss implementation issues with developer 
staff and among themselves. Developer staff were also available by phone and email on 
an as-needed basis. Partners for Literacy assigned a consultant to each of their projects. 
These consultants conducted the mentoring visits and stayed in regular contact with 
assigned projects by email and phone. 



Instructional Practices and Fidelity in CLIO Classrooms 

Two kinds of data on classrooms were collected to describe the types and 
quality of the instructional practices. First, observations were conducted of all 
classrooms in the sample, both treatment and control, to document the extent to which 
the children were exposed to instructional practices shown in previous research to be 
related to children's emergent literacy skills. Second, in only the treatment classrooms. 
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observations were conducted to evaluate the fidelity of implementation of the CLIO 
curricula. Since the CLIO curricula were chosen because they represented research- 
based instructional practices for supporting emergent literacy, it is highly likely that the 
CLIO curricula overlap substantially (1) with the universe of potentially effective early 
childhood education practices measured in the CLIO observations and (2) with each 
other. 



Figure 3-1 represents this pictorially. In the figure, the smaller circles 
represent the instructional practices used by the two CLIO preschool curricula (as 
intended), while the larger circle represents all of the instructional practices that 
research has shown are linked to child literacy outcomes. The overlap between each of 
the smaller circles and the larger circle represents the extent to which each of the CLIO 
curricula embody the "best" practices as defined by the field. Further, the two smaller 
circles themselves overlap to a large degree, since the two CLIO preschool curricula use 
some common instructional practices across their multiple activities. 



Fidelity of Implementation 

Program fidelity is the degree of fit between the developer-defined 
components of the curriculum and its actual implementation in the classroom. In other 
words, how closely does the classroom instruction match the specifications of the 
curriculum that were set out in the curriculum materials and training? Research studies 
often measure fidelity because of the added value that an understanding of fidelity can 
bring to interpreting impacts of a curriculum. If a curriculum is poorly implemented, 
then any negative findings about that curriculum might indicate directions for 
curriculum modification or modification to teacher training procedures. 



53 




Figure 3-1. Relationship of Measuring Fidelity of Implementation and Overall 
Instructional Practices 




Measures, of Fidelity oF 
two curricula... 

■3 Overlap a great ceal 
with each uther bul are 
not ine same 

© Overlap a great deal 
wilh measures of overall 
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curriculum alone 



The study measured the fidelity of implementation of the CLIO curricula in 
two ways: evaluation staff conducted independent observations of preschool and 
parenting classrooms, and the curriculum developers rated the level of implementation 
of both the preschool curriculum and parenting curriculum in each project. The rating 
system was the same for both sources of information. A 5-point Likert scale was used to 
rate each project's implementation of the curriculum as one of the following: 

1. not appreciably implemented, 

2. partially implemented, 

3. half-way implemented, 

4. almost fully implemented, or 

5. fully implemented. 

Ratings by Independent Observers. The first set of fidelity ratings was at 
the classroom level and was based on the classroom observations conducted by trained 
independent observers. The observations were conducted in the spring of each 
implementation year, which yielded two fidelity ratings by independent observers over 
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the life of the study. 18 These corresponded to approximately 9 months and 20 months 
after training. 

During the classroom observations, the observers completed fidelity 
checklists that were developed by the evaluation team with input from the developers 
themselves. The checklists were designed to closely match the specific 
recommendations made to teachers during summer trainings about how to put the 
curriculum into place in their classrooms. Separate checklists were designed for 
preschool and parenting classrooms, and observers rated them separately. (See 
appendix G for more detail on the development of the CLIO fidelity checklists and 
scoring rubrics.) 

Ratings by Curriculum Developers. Each curriculum developer also rated 
the fidelity of implementation of the curriculum in each of their projects. Developers 
assigned their ratings at the project level. Fidelity was rated separately for the preschool 
curriculum and the parenting curriculum, and separately for each of the 2 
implementation years of the CLIO study. Further, in each implementation year, the 
developers provided fidelity ratings twice, in the winter and the spring. Thus, the 
developers provided fidelity ratings of their projects four times over the life of the 
study, which described fidelity approximately 6 months, 11 months, 18 months, and 23 
months after the initial training. According to developers, their fidelity ratings included 
information gained through records received from implementing projects, one-on-one 
emails and phone calls between developer and project staff, and coaching visits to 
projects by developer staff. (See appendix G for further information on fidelity 
measurement.) 

Summary of Ratings. On average, curriculum implementation was rated as 
having reached about 50 percent of what represented full implementation of the models 
(or "3" on the 5-point rating scale). The average fidelity ratings were consistent across 
the two sets of raters (see table 3-1 for average ratings by the observers and table 3-2 for 
average ratings by the developers). 



18 Four projects refused to implement their randomly assigned curriculum in one year or the other. 
Despite this refusal, all four projects did allow continued data collection, including classroom 
observations and observer fidelity measurement. 
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Table 3-1. Average Classroom-Level Fidelity Ratings 3 by Study Group and Classroom 
Type for Spring 2005 and Spring 2006: Independent Observer Ratings 









Study group 






All study 
groups 1 


Let's Begin 
with the 
Letter 
People 
(ECE) 


Let's Begin 
and Play & 
Learning 
Strategies 
(ECE/PE) 


Partners 

for 

Literacy 

(ECE) 


Partners 

for 

Literacy 

(ECE/PE) 


Early childhood education (ECE) 












2005 


2.84 


3.38 


3.39 


2.20 


2.49 


2006 


2.85 


3.61 


3.55 


2.00 


2.20 


Parenting education (PE) 


2005 


3.30 


t 


3.28 


t 


3.32 


2006 


3.46 


t 


3.52 


t 


3.40 



a The fidelity ratings range from 1 (not appreciably implemented) to 5 (fully implemented). 
f Not applicable. 

1 For the ECE classroom rating, "all study groups" is the average rating across all four study groups. For the PE 
classroom rating, "all study groups" is the average rating for the two study groups with PE components. 

NOTES: The data are based on classrooms with at least one child who took a spring CLIO assessment in the 
relevant year. Observer fidelity ratings were conducted at the classroom level as part of the larger set of 
classroom observations of instructional practices. 

SOURCE: U.S. Department of Education, Even Start Classroom Literacy Interventions and Outcomes Study, 
"Observer Fidelity Ratings," Spring 2005 and Spring 2006. 
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Table 3-2. Average Project-Level Fidelity Ratings 3 by Study Group and Classroom 
Type for Spring 2005 and Spring 2006: Developer Ratings 









Study group 






All study 
groups 1 


Let's Begin 
with the 
Letter 
People 
(ECE) 


Let's Begin 
and Play & 
Learning 
Strategies 
(ECE/PE) 


Partners 

for 

Literacy 

(ECE) 


Partners 

for 

Literacy 

(ECE/PE) 


Early childhood education (ECE) 












2005 


2.99 


3.30 


3.72 


2.46 


2.45 


2006 


3.29 


4.10 


3.33 


2.59 


3.05 


Parenting education (PE) 


2005 


2.96 


t 


3.29 


t 


2.59 


2006 


3.26 


t 


3.41 


t 


3.10 



a The fidelity ratings range from 1 (not appreciably implemented) to 5 (fully implemented). 
f Not applicable. 

1 For the ECE project rating, "all study groups" is the average rating across all four study groups. For the PE 
project rating, "all study groups" is the average rating for the two study groups with PE components. 

NOTES: Developer fidelity ratings were recorded at the project level. In projects with more than one ECE or PE 
class, the fidelity scores for classrooms were averaged. 

SOURCE: U.S. Department of Education, Even Start Classroom Literacy Interventions and Outcomes Study, 
"Developer Fidelity Ratings," Spring 2005 and Spring 2006. 



A visual inspection of the fidelity data suggests that fidelity ratings varied by 
curriculum and by implementation year, as follows (tables 3-1 and 3-2). 

• For both observer ratings and developer ratings, the fidelity of 
implementation for the projects implementing Let's Begin with the 
Letter People and PALS were generally higher than those for the 
Partners for Literacy projects. 

• For both observer ratings and developer ratings, the average fidelity 
ratings were higher in the second year of implementation of the CLIO 
curricula. This was the expected pattern, since fidelity after nearly 2 
years of experience implementing the curricula (2006) should be higher 
than fidelity after only a single year of implementation (2005). The one 
exception was the observer ratings of the fidelity of implementation in 
the Partners for Literacy preschool classrooms, which were lower in 
2006 compared with 2005. 
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In 2005, across all study groups, observers rated 24 percent of preschool 
classrooms as achieving "4" or higher on the 5-point fidelity scale and 42 percent of 
parenting education classrooms as achieving "4" or higher on the fidelity scale (figure 
3-2). In 2006, observers rated 29 percent of preschool classrooms and 60 percent of 
parenting education classrooms as achieving at least "4" on the 5-point fidelity scale. 

Figure 3-2. Percentage Distribution of Fidelity Ratings of CLIO Treatment 
Classrooms by Classroom Type for Spring 2005 and Spring 2006: 
Observer Ratings 




NOTES: The fidelity ratings range from 1 (not appreciably implemented) to 5 (fully implemented). The 
ECE rating includes classrooms in all four study groups, since all four had CLIO ECE curricula. The PE 
rating includes classrooms in the two study groups with CLIO PE curricula. The data are based on 
classrooms with at least one child who took a CLIO assessment in the relevant year. Detail may not sum 
to total due to rounding. 

SOURCE: U.S. Department of Education, Even Start Classroom Literacy Interventions and Outcomes 
Study, "Observer Fidelity Ratings," Spring 2005 and Spring 2006. 

Across all study groups in 2005, developers rated 33 percent of projects as 
achieving at least "4" on the 5-point fidelity scale for the preschool curricula and rated 
37 percent of projects as achieving at least "4" on the fidelity scale for parenting 
curricula (figure 3-3). In 2006, developers rated 38 percent of projects as "4" or above on 
the fidelity scale for preschool curricula and 44 percent of projects as "4" or above on 
the fidelity scale for parenting curricula. 
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Figure 3-3. Percentage Distribution of Fidelity Ratings of CLIO Treatment Projects 
by Classroom Type for Spring 2005 and Spring 2006: Developer Ratings 




NOTES: The fidelity ratings range from 1 (not appreciably implemented) to 5 (fully implemented). Any 
rating that fell between two intervals was rounded down for the graph. The ECE rating includes projects 
in all four study groups, since all four had CLIO ECE curricula. The PE rating includes projects in the two 
study groups with CLIO PE curricula. Detail may not sum to total due to rounding. 

SOURCE: U.S. Department of Education, Even Start Classroom Literacy Interventions and Outcomes 
Study, "Developer Fidelity Ratings," Spring 2005 and Spring 2006. 



Exposure to the Curricula 

Another important component of the implementation of any curriculum is 
exposure. In order to benefit from the curricula in significant ways, children and 
parents would need sufficient exposure to the curricula. Even Start guidelines do not 
specify an expected level of exposure for children or parents, and the hours of 
instruction offered by local projects vary widely. Curriculum developers were aware of 
this at the outset of the study, and the curricula were designed to accommodate this 
variation. To measure exposure to services in general and to the CLIO curricula, we 
collected monthly participation data from all CLIO projects (both treatment and control) 
on the number of hours of instruction for each preschooler and the number of hours of 
parenting instruction for each parent. 
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Preschool Participation. Projects reported that they offered an average of 80 
hours of preschool education per month in both years of the study. 19 The participation 
data show that in the first 7 months of the school year, 20 children received an average of 
50 hours of instruction per month. 21 Almost three-quarters of children attended 
preschool 60 hours or fewer per month, where 60 hours per month is the equivalent of a 
half-day program meeting 5 days a week (see figure 3-4). In approximately half of CLIO 
projects, no children participated in preschool education for 60 hours or more per 
month. This is consistent with participation data from the Third National Even Start 
Evaluation (St.Pierre et al. 2003). 

Participation in Parenting Education. During the 2 implementation years, 
projects reported that they offered parents a monthly average of 14 hours of PE and 11 
hours of PC. 22 As was true for children and the preschool curricula, parents also 
received limited exposure to the parenting curricula. Over the 7-month period 
preceding spring data collection, parents received an average of 13 hours combined 
parenting instruction per month. 23 The majority of parents (over 80 percent) had 20 or 
fewer hours of combined parenting instruction per month (see figure 3-5). This is 
consistent with participation data from the Third National Even Start Evaluation 
(St.Pierre et al. 2003). 



19 Preschool monthly hours offered ranged from 24 to 160 hours, with a standard deviation of 31.1. 

20 Wc use 9-month participation as an outcome, but for purposes of understanding the intervention, the 7- 
month figures are more useful. Clearly participation in April and May cannot affect performance in 
March assessments. 

21 Monthly preschool participation hours ranged from 0 to 152 hours, with a standard deviation of 31.5. 

22 PE monthly hours offered ranged from 4 to 40 hours, with a standard deviation of 6.6. PC monthly 
hours offered ranged from 2 to 28 hours, with a standard deviation of 5.4. 

23 Monthly PE/PC participation hours ranged from 0 to 81 hours, with a standard deviation of 9.4. The 
high figure includes three parents for whom Even Start projects reported more than 68 hours per 
month, on average, of parent participation. Excluding these outliers, the monthly PE/PC participation 
hours ranged from 0 to 64, with a standard deviation of 9.2 hours. 
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Figure 3-4. Distribution of CLIO Children by Average Monthly Hours of ECE 
Participation, Spring 2005 and Spring 2006 




NOTE: Average monthly hours is calculated over the 7-month period of September through March. The 
data were run on children in the combined 2005/2006 analysis sample. 

SOURCE: U.S. Department of Education, Even Start Classroom Literacy Interventions and Outcomes 
Study, "Instructional Services Participation Form (ISPF)," 2004-05 and 2005-06. 
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Figure 3-5. Distribution of CLIO Parents by Average Monthly Hours of PE/PC 
Participation, Spring 2005 and Spring 2006 




Average PE/PC hours per month 



NOTES: Average monthly hours is calculated over the 7-month period of September through March. The 
data are parent-level runs of children in the combined 2005/2006 analysis sample. 

SOURCE: U.S. Department of Education, Even Start Classroom Literacy Interventions and Outcomes 
Study, "Instructional Services Participation Form (ISPF)," 2004-05 and 2005-06. 



Control Projects During CLIO Study 

While the CLIO curricula were being implemented in the four treatment 
groups, projects in the control group continued with their usual services in the four 
Even Start instructional areas. Understanding the control classrooms can help us 
interpret the impacts on parents and children. 

The study has two sources of information about the instruction in the control 
classes. First, project directors in all of the CLIO projects, including control projects, 
were asked about any formal curricula used in their preschool or parent education 
programs. Second, the OMLIT observations provide detailed information on the 
instructional processes in both control and treatment projects. We consider the OMLIT 



62 






data to provide a richer and more valid estimate of the differences between the 
treatment and control projects than reports from project directors. 

Based on project director reports of curricula used, we estimated that most 
control projects used a formal early childhood curriculum and about half used a formal 
parenting curriculum. Specifically, in both implementation years a majority of control 
projects reported using at least one formal preschool curriculum — 71 percent in 2005 
and 76 percent in 2006. Among the control projects that reported using a curriculum, 
the two most commonly used were High Scope and Creative Curriculum. About half of 
the control projects reported using one of these two curricula in each of the 
implementation years. Other curricula cited included a range of instructional 
programming, including published comprehensive curricula, literacy-focused 
supplemental curricula, informal non-published curricula, and local or other reading 
initiatives. 



In both implementation years, over half of control projects reported using at 
least one formal PE curriculum — 58 percent in 2005 and 64 percent in 2006. Parents as 
Teachers was the only formal curriculum named by more than a few projects. A 
number of other curricula were named by individual projects (e.g., MotherRead, the 
Bowdoin method), but most projects used materials or activities developed by the 
project either on its own or based on materials available to the field. 

The OMLIT observations data from spring 2005 and spring 2006 were used 
to create a set of variables that provide a descriptive profile of both treatment and 
control classrooms (table 3-3). As a group, the control classrooms spent about 15 percent 
of the day in literacy-related activities, including reading, alphabet/print knowledge, 
phonological awareness, and emergent writing. For treatment classrooms, time spent in 
literacy-related activities ranged from 16 to 19 percent of the day. Including these 
literacy activities, control classrooms spent about 45 percent of the day in activities that 
are often considered by developmental psychologists to have particularly high value for 
children because of the opportunities for children to construct knowledge and receive 
feedback on their interactions with materials, peers, and adults in the classroom (Bruner 
and Watson 1983). Treatment classrooms spent from about 42 to 51 percent of the day in 
these high-value activities. The remainder of the day for all classrooms was spent in 
daily group activities, including review of the calendar, weather, and attendance; gross 
motor play and transition; and meals/snacks. 
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